Connect with us


Why success brings its own challenges in IT: Fixing the real-time problems of real-time event streaming with an event portal

Spokesperson: Tom Fairbairn, Distinguished Engineer, Solace

Amidst the recent hype around generative AI, there is a perhaps quieter technology revolution going on: the increasing adoption of event-driven architecture (EDA), an example of which is “event streaming” across the enterprise. Regardless of industry, (manufacturing, retail or even HR, amongst others), adoption of EDA is increasing use cases across the board as organisations realise the value of EDA. From decoupling applications, the ability to be more responsive to internal and external user requests, and reducing operational costs, EDA ultimately allows enterprises the freedom to become more agile and real-time in their day-to-day activities.

A recent IDC Infobrief found that of those who have deployed EDA across multiple use cases, 93% say it either met or exceeded their expectations. Going even further, 82% of those surveyed have stated their plans to apply EDA across their company for a further two/three use cases, within the next 24 months.

Increasing adoption and time-to-market scrambles, along with bigger data flows and more complex deployments lead to difficulties in getting real visibility across a complex, and often rapidly scaling ecosystem of events.

As a result, some are struggling to realise the best return on their EDA investment. The issue can sometimes compound, as an organisation may have reached a stage of maturity regarding multiple streaming use cases, but are left with siloed brokers, clusters, topics and schematics weaved on top of each other. A layer cake effect occurs in the architecture, making it difficult to truly see where they can optimise the information insights chain.

Growing pains are a natural part of scalability, and EDA is no exception.

Three pain points typically arise:

1-Poor visibility requires a clear and clean portal

One of the unsung heroes of EDA is application decoupling, which aids flexibility and agility. However, this can often be detrimental to visibility, making changes to existing applications difficult and risky. This lack of visibility leads to confused ownership, as event producers don’t know who the consumers are, and consumers don’t know who the publishers are. Some event streaming systems, such as Kafka, rely on static or near-static topics which can be abandoned or duplicated.

This directly affects the end-user experience. Say a person is using the application every day and wants to add an extra attribute to the data – they’d have no direct line to contact because they can’t see who produced the data.

An event portal’s single window into the structure of an EDA benefits the event-streaming ecosystem by acting as a native discovery agent to quickly produce visual aids to help locate consumers and producers. Architects and administrators on the business side are able to view the relationship between producers and consumers, as well as their event interface and KPIs, including their most and least-used topics.

Another issue that occurs when scaling infrastructure and information flows is the inability to fully grasp the downstream impact of changes made to the architecture. Evolving and architecture will require visibility on whether microservices are affected by a given event before even deploying a new feature or function, to ensure it doesn’t bring down that system. This must be done sometimes in seconds. While real-time is part of the solution, an event portal can automate the scanning of an internal system, helping visualise a complete view of endpoints and event streams, rather than making the business manually draw them between microservices. For those looking to better understand their infrastructure, particularly as they scale up, an event portal helps deploy new services faster.

2-Event Streaming is not a one-and-done – the issue with limited sharing and reuse

Data has been the new oil for quite some time, and much like oil, when one needs to see data, they often need it right there and then. Real-time data is the most valuable data that can exist within your organisation, but if it is siloed in one specific department, overall business decision-makers don’t know about it, and that means it is not providing its full potential.

Developers must understand that their hard work won’t be realised until they can catalogue all that data and the benefits that come with it. Otherwise, it may get stale and quickly become out of date.

An event portal can create a real-time catalogue of event data, able to list all topics, event streams and schemas for each application, as well as the owner and best point of contact. Not only does this speed up internal developments by letting developers share, discover, and re-use existing event-streaming assets, but they can do so outside of the organisation so that customers and partners can benefit from it too – an often neglected consideration when technologies such as Kafka focus on operational management.

3-Security and governance in event streams

The fact that event-driven systems are dynamic and decentralised is a major selling point. However, that can run the risk of unique security challenges.

A trade-off that commonly occurs when implementing event streaming is that, as they include access control rules, developers tend to be too permissive to ensure agility, creating a lack of visibility or proper cataloguing of the data. With time, that runs the risk of interfering with business data security, governance and compliance. The issue only gets worse over time as the use of data evolves across the applications, with each new human addition.

With a thinning attack surface and an increase in never-before-seen malware variants, security in business must pivot from reactive to proactive.

An event portal helps users organise systems into application domains, create and import payload schema definitions in a variety of formats and better define event interactions. This improves IT governance by more easily allowing control for who can access which resources while giving those the ability to create and track every version of EDA objects as they evolve.

By putting visibility at the forefront, administrators can spend less time worrying about ensuring their security is in line with internal and regulatory policies. Outside of real-time visibility, the true value of event-driven architecture lies in the ability to easily spot and re-use all of these data assets. However, that can only be done if they are documented, managed and governed properly. The event portal is necessary to solve this, acting as a discovery tool, de facto guide, and being able to dictate the more effective way to manage event streams.

Real-time event streams and the success of powerful architectures have brought with them the need to address the rapidly growing complexity of event streaming estates. Single, multi-broker event portal technology is the solution to helping organisations discover, document, govern and manage the lifecycle of their real-time event streams across the enterprise, and it will become increasingly important as more organisations embrace EDA as a foundational platform.

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *


How can a payments strategy support business growth?

Source: Finance Derivative

Following the global economic upheaval brought on by the pandemic, businesses are once again prioritising growth on a global scale. While every business recognises the importance of expansion, their methods, obstacles, and risks differ greatly.

In the following article, Sonya Geelon, Chief Commercial Officer at Conferma, explores some of the most common challenges holding businesses back, and how by including innovative payments solutions in your payment strategy, you can successfully position your business to expand into global markets.

Barriers to global expansion

At Conferma, we wanted to know what businesses felt stood between them and their growth ambitions, so we spoke to 400 financial decision makers to find out.

The research, shared in our new Growth Ignition Index report, identified global expansion as a key priority for businesses looking to grow across all regions. Significant drivers included increasing customer demand (46 per cent), maintaining a consistent cashflow (36 per cent) and undertaking digital transformation (34 per cent.) Businesses also highlighted a number of barriers, such as identifying valuable markets to expand into (27 per cent) and navigating complex cross-border payment systems (13 per cent.) The following sheds light on some of the factors that businesses perceive to be hindering their growth.

Operational inefficiencies

It’s a well-known fact that operational efficiency is crucial for giving businesses the competitive edge. If your processes run smoothly and effectively, you’re likely in a good position to grow. However, a third (33 per cent) of businesses identified operational inefficiencies as a significant sticking point, particularly among small-and-medium sized organisations. This perhaps indicates that larger companies have already invested in boosting efficiency to a degree, however, the issue was noted across businesses of all sizes.

Complex cross-border payments

Successful growth relies heavily on being able to make fast, seamless transactions, however, recent research from Rapyd found that 38 per cent of businesses experience delays of five days or more when sending or receiving international payments.[1] Costs and delays in cross-border transactions can have a significant impact on growth, cutting into revenues, restricting cash flow and complicating financial planning. Our own research highlighted this, with 14 per cent of businesses reporting slow and/or complex cross-border payments as a significant barrier to expansion.

So how can businesses overcome these challenges and unlock global growth?

Taking your payments strategy virtual

Amid the array of payment options available in the market, virtual cards have emerged as a versatile solution, valued by users globally. According to Juniper Research, the global value of virtual cards will increase over threefold in just 5 years, climbing from $1.9 trillion in 2021 to a staggering $6.8 trillion by 2026.[2]

So how do they work?

Virtual cards are essentially digital versions of traditional credit cards. The technology generates a 16-digit card  number, allowing an employee to make payments without having to physically hand over a card. Instead, they provide the virtual card number, expiration date, and security code, just like they would with a regular credit or debit card.

Virtual cards come with built-in fraud and security features, enabling restrictions on usage. For instance, users can set a specific date range or limit usage to certain merchants. This ensures that any attempts to exceed the set amount, use the card at unauthorised merchants, or spend beyond the specified date range will result in a declined transaction.

Using a virtual card provider allows access to extensive, pre-existing payments ecosystems. For example, Conferma connects 75+ card issuers and banks across the world. This enables businesses to use virtual cards in 62 different currencies, making international payments frictionless while mitigating costly cross-border fees. Virtual cards can also help boost cashflow and improve operational efficiency, automating reconciliation and cutting lengthy processing times. By removing convoluted payment processes, virtual cards give businesses the freedom to grow in the markets they deem most valuable, not just most accessible.

Of those surveyed, four out of five  respondents (82 per cent) plan on expanding their virtual card usage in the next twelve months, with 64 per cent extending usage to additional payment needs. Businesses already using virtual cards also anticipate a substantial increase in the volume of payments they make virtually, with our data projecting a rise from 45 to 57 per cent of all payments being made using virtual cards in the next 12 months.

Virtual cards offer a compelling solution to the challenges limiting international growth by offering enhanced security, streamlined operational processes, and seamless cross-border transactions. By embracing virtual cards as a strategic tool, organisations can unlock opportunities for growth and innovation, empowering them to navigate the complexities of international commerce with ease.

[1] The 2023 State of Cross-Border Payments, Rapyd, 2023.

[2] Virtual Cards: B2B and B2C Applications, Competitive Analysis & Market Forecasts 2021-2026, Juniper Research

Continue Reading


How can businesses make the cloud optional in their operations?

Max Alexander, Co-founder at Ditto

Modern business apps are built to be cloud-dependent. This is great for accessing limitless compute and data storage capabilities but when connection to the cloud is poor or shuts down, business apps stop working, impacting revenue and service. If real-time data is needed for quick decision-making in fields like healthcare, a stalled app can potentially put people in life-threatening situations.

Organisations in sectors as diverse as airlines, fast food retail, and ecommerce that have deskless staff who need digital tools accessible on smartphones, tablets and other devices to do their jobs. But because of widespread connectivity issues and outages, these organisations are beginning to consider how to ensure these tools can operate reliably when the cloud is not accessible. 

The short answer is that building applications with a local-first architecture can help to ensure that they remain functional when disconnected from the internet. But then, why are not all apps built this way? The simple answer is that building and deploying cloud-only applications is much easier as ready-made tools for developers help expedite a lot of the backend building process. The more complex answer is that a local-first architecture solves the issue of offline data accessibility but does not solve the critical issue of offline data synchronisation. Apps disconnected from the internet still have no way to share data across devices. That is where peer-to-peer data sync and mesh networking come into play.

Combining offline-first architecture with peer-to-peer data sync

In the real world, what does an application like this look like?

  • Apps must prioritise local data sync. Rather than sending data to a remote server, applications must be able to write data using its local database in the first instance, and then listen for changes from other devices, and recombine them as needed. Apps should utilise local transports such as Bluetooth Low Energy (BLE) and Peer-to-Peer WiFi (P2P Wi-Fi) to communicate data changes in the event that the internet, local server, or the cloud is not available.
  • Devices are capable of creating real-time mesh networks. Nearby devices should be able to discover, communicate, and maintain constant connections with devices in areas of limited or no connectivity.
  • Seamlessly transition from online to offline (and vice versa). Combining local sync with mesh networking means that devices in the same mesh are constantly updating a local version of the database and opportunistically syncing those changes with the cloud when it is available.
  • Partitioned between large peer and small peer mesh networks to not overwhelm smaller networks if they try to sync every piece of data. In order to do this, smaller networks will only sync the data that it requests, so developers have complete control over bandwidth usage and storage. This is vital when connectivity is erratic or critical data needs prioritising. Whereas, the larger networks sync as much data as they can, which is when there is full access to cloud-based systems.
  • Ad-hoc to enable devices to join and leave the mesh when they need to. This also means that there can be no central server other devices are relying on.
  • Compatible with all data at any time. All devices should account for incoming data with different schemas. In this way, if a device is offline and running an outdated app version, for example, it still must be able to read new data and sync.

Peer-to-peer sync and mesh networking in practice

Let us take a look at a point-of-sale application in the fast-paced environment of a quick-service restaurant. When an order is taken at a kiosk or counter, that data must travel hundreds of miles to a data centre to arrive at a device four metres away in the kitchen. This is an inefficient process and can slow down or even halt operations, especially if there is an internet outage or any issues with the cloud.

A major fast-food restaurant in the US has already modernised its point of sale system using this new architecture and created one that can move order data between store devices independently of an internet connection. As such, this system is much more resilient in the face of outages, ensuring employees can always deliver best-in-class service, regardless of internet connectivity.

The vast power of cloud-optional computing is showcased in healthcare situations in rural areas in developing countries. By using both peer-to-peer data sync and mesh networking, essential healthcare applications can share critical health information without the Internet or a connection to the cloud. This means that healthcare workers in disconnected environments can now quickly process information and share it with relevant colleagues, empowering faster reaction times that can save lives.

Although the shift from cloud-only to cloud-optional is subtle and will not be obvious to end users, it really is a fundamental paradigm shift. This move provides a number of business opportunities for increasing revenue and efficiencies and helps ensure sustained service for customers.

Continue Reading


When something personal fills an important gap in the market 

by Cécile Mazuet-Eller, founder of NameSwitch

There aren’t many business ideas that go from a personal experience to filling an important gap in the market. However, this is certainly the case for NameSwitch, the UK’s pioneering and only name changing support service launched in 2018. But what inspired its inception and what challenges did it face? Here, Cécile Mazuet-Eller, the founder of the company, in its seventh year, explains.

My entrepreneurial journey is a bit unusual in that it started from my own experience of going through a divorce, which became a pivotal turning point for me not only emotionally, but practically too. I wanted to remove my married name, and I had a visceral reason to do so as I really didn’t want to keep it. Feeling extremely frustrated at still receiving letters and official documents featuring my previous name, I was desperate to change it but like for so many people it became a stop-start, arduous task.

Once I started the process, I realised it was taking up far too much time I didn’t have; being a single mum to two young children and working full-time is no mean feat, so when I embarked on the name changing process I realised it wasn’t going to be easy.  Searching for a solution to help, all I came up with was a service covering the US and Canada, but nothing that worked for the UK, so in the end, I spent a whole year to get everything changed that had to be, which proved long and stressful to say the least.

Nurturing the idea

In the early days I was fortunate enough to be surrounded by positive people who had good contacts, and who saw the viability of my idea. Living in a small community filled with intelligent and well-rounded people, I wasn’t short of encouragement from them and friends, who recognised as well as I did there was a definite gap in the market. Working with a web development team in Serbia which was also recommended, I enlisted additional help from a university student on some research.

I always wanted to run my own business, and there were several reasons why I needed to embark on something new. As the only breadwinner in the house, there were mounting bills while balancing the demands of motherhood and other financial responsibilities. Cash was limited but what little I had was used carefully which I put into the business.

In the early stages, which included the development of the unique technology that underpins the service, I carved pockets of time at night and on weekends to create a strong foundation for the business. Creating something completely from scratch was like a form of healing, which is why it was and remains such a personal project.

Mulling over the idea for at least two years following the original lightbulb moment, the business was registered in 2015, with time needed for building the robust platform in order to  create a viable product. Drawing on my previous experience, I investigated overseas equivalents, financials and marketing intelligence ensuring there was a genuine need for the service in the UK. Fortunately enough I was able to share my plans with my employer at the time, who turned out to be my biggest supporters, becoming my first paying customer who purchased a NameSwitch for his ex-wife, who was getting married to someone else!

With a career in telecommunications and a degree in marketing, I was already used to hard work and having the support and encouragement from my telecoms team was extremely helpful.   

Support and coaching

Coaching was an important element of the start-up process, obtained through a wider network and some financial support from family,  with no other funding or investment being available.

The challenges

Presented with certain obstacles like all businesses are, there was a lot to juggle and at times it felt like too much but I managed to navigate the complexities involved. When Covid hit that was a huge set-back, given that our biggest target market was and still is, newly-weds. With all weddings being banned, it hit NameSwitch hard, but our saving grace were the people who used the time to change their name’s in lockdown, by doing something they previously didn’t have time for. Being 100% employed by the business by this stage, it turned into a year of survival and another big challenge.  

In 2022-2023 we concentrated on growth for NameSwitch, when me and my dedicated team were satisfied with the service, it was time to consider investment into PR, advertising and partnerships to increase brand awareness to reach the revenues that were needed.

In 2022-2024, it was forecast that 285,000 – 415,000 weddings will take place resulting from the pandemic, which has reflected well on the business in recent years. And amidst the trials and tribulations it’s proved to be both exhilarating and exhausting in equal measure.

With hindsight, there are certain things I’d have done differently, such as bringing in a partner early on to put us in a stronger position sooner, and adding more resource  to improve growth, but I know that’s all part of the steep learning curve and something to take with me to projects in the future.

Advice for aspiring entrepreneurs

For anyone contemplating their own entrepreneurial endeavours, I’d recommend to ‘one hundred percent go for it’ – but do not bet the house on it and whatever happens, embrace the journey.

Continue Reading

Copyright © 2021 Futures Parity.