Connect with us

Business

Why success brings its own challenges in IT: Fixing the real-time problems of real-time event streaming with an event portal

Spokesperson: Tom Fairbairn, Distinguished Engineer, Solace

Amidst the recent hype around generative AI, there is a perhaps quieter technology revolution going on: the increasing adoption of event-driven architecture (EDA), an example of which is “event streaming” across the enterprise. Regardless of industry, (manufacturing, retail or even HR, amongst others), adoption of EDA is increasing use cases across the board as organisations realise the value of EDA. From decoupling applications, the ability to be more responsive to internal and external user requests, and reducing operational costs, EDA ultimately allows enterprises the freedom to become more agile and real-time in their day-to-day activities.

A recent IDC Infobrief found that of those who have deployed EDA across multiple use cases, 93% say it either met or exceeded their expectations. Going even further, 82% of those surveyed have stated their plans to apply EDA across their company for a further two/three use cases, within the next 24 months.

Increasing adoption and time-to-market scrambles, along with bigger data flows and more complex deployments lead to difficulties in getting real visibility across a complex, and often rapidly scaling ecosystem of events.

As a result, some are struggling to realise the best return on their EDA investment. The issue can sometimes compound, as an organisation may have reached a stage of maturity regarding multiple streaming use cases, but are left with siloed brokers, clusters, topics and schematics weaved on top of each other. A layer cake effect occurs in the architecture, making it difficult to truly see where they can optimise the information insights chain.

Growing pains are a natural part of scalability, and EDA is no exception.

Three pain points typically arise:

1-Poor visibility requires a clear and clean portal

One of the unsung heroes of EDA is application decoupling, which aids flexibility and agility. However, this can often be detrimental to visibility, making changes to existing applications difficult and risky. This lack of visibility leads to confused ownership, as event producers don’t know who the consumers are, and consumers don’t know who the publishers are. Some event streaming systems, such as Kafka, rely on static or near-static topics which can be abandoned or duplicated.

This directly affects the end-user experience. Say a person is using the application every day and wants to add an extra attribute to the data – they’d have no direct line to contact because they can’t see who produced the data.

An event portal’s single window into the structure of an EDA benefits the event-streaming ecosystem by acting as a native discovery agent to quickly produce visual aids to help locate consumers and producers. Architects and administrators on the business side are able to view the relationship between producers and consumers, as well as their event interface and KPIs, including their most and least-used topics.

Another issue that occurs when scaling infrastructure and information flows is the inability to fully grasp the downstream impact of changes made to the architecture. Evolving and architecture will require visibility on whether microservices are affected by a given event before even deploying a new feature or function, to ensure it doesn’t bring down that system. This must be done sometimes in seconds. While real-time is part of the solution, an event portal can automate the scanning of an internal system, helping visualise a complete view of endpoints and event streams, rather than making the business manually draw them between microservices. For those looking to better understand their infrastructure, particularly as they scale up, an event portal helps deploy new services faster.

2-Event Streaming is not a one-and-done – the issue with limited sharing and reuse

Data has been the new oil for quite some time, and much like oil, when one needs to see data, they often need it right there and then. Real-time data is the most valuable data that can exist within your organisation, but if it is siloed in one specific department, overall business decision-makers don’t know about it, and that means it is not providing its full potential.

Developers must understand that their hard work won’t be realised until they can catalogue all that data and the benefits that come with it. Otherwise, it may get stale and quickly become out of date.

An event portal can create a real-time catalogue of event data, able to list all topics, event streams and schemas for each application, as well as the owner and best point of contact. Not only does this speed up internal developments by letting developers share, discover, and re-use existing event-streaming assets, but they can do so outside of the organisation so that customers and partners can benefit from it too – an often neglected consideration when technologies such as Kafka focus on operational management.

3-Security and governance in event streams

The fact that event-driven systems are dynamic and decentralised is a major selling point. However, that can run the risk of unique security challenges.

A trade-off that commonly occurs when implementing event streaming is that, as they include access control rules, developers tend to be too permissive to ensure agility, creating a lack of visibility or proper cataloguing of the data. With time, that runs the risk of interfering with business data security, governance and compliance. The issue only gets worse over time as the use of data evolves across the applications, with each new human addition.

With a thinning attack surface and an increase in never-before-seen malware variants, security in business must pivot from reactive to proactive.

An event portal helps users organise systems into application domains, create and import payload schema definitions in a variety of formats and better define event interactions. This improves IT governance by more easily allowing control for who can access which resources while giving those the ability to create and track every version of EDA objects as they evolve.

By putting visibility at the forefront, administrators can spend less time worrying about ensuring their security is in line with internal and regulatory policies. Outside of real-time visibility, the true value of event-driven architecture lies in the ability to easily spot and re-use all of these data assets. However, that can only be done if they are documented, managed and governed properly. The event portal is necessary to solve this, acting as a discovery tool, de facto guide, and being able to dictate the more effective way to manage event streams.

Real-time event streams and the success of powerful architectures have brought with them the need to address the rapidly growing complexity of event streaming estates. Single, multi-broker event portal technology is the solution to helping organisations discover, document, govern and manage the lifecycle of their real-time event streams across the enterprise, and it will become increasingly important as more organisations embrace EDA as a foundational platform.

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Business

How can businesses make the cloud optional in their operations?

Max Alexander, Co-founder at Ditto

Modern business apps are built to be cloud-dependent. This is great for accessing limitless compute and data storage capabilities but when connection to the cloud is poor or shuts down, business apps stop working, impacting revenue and service. If real-time data is needed for quick decision-making in fields like healthcare, a stalled app can potentially put people in life-threatening situations.

Organisations in sectors as diverse as airlines, fast food retail, and ecommerce that have deskless staff who need digital tools accessible on smartphones, tablets and other devices to do their jobs. But because of widespread connectivity issues and outages, these organisations are beginning to consider how to ensure these tools can operate reliably when the cloud is not accessible. 

The short answer is that building applications with a local-first architecture can help to ensure that they remain functional when disconnected from the internet. But then, why are not all apps built this way? The simple answer is that building and deploying cloud-only applications is much easier as ready-made tools for developers help expedite a lot of the backend building process. The more complex answer is that a local-first architecture solves the issue of offline data accessibility but does not solve the critical issue of offline data synchronisation. Apps disconnected from the internet still have no way to share data across devices. That is where peer-to-peer data sync and mesh networking come into play.

Combining offline-first architecture with peer-to-peer data sync

In the real world, what does an application like this look like?

  • Apps must prioritise local data sync. Rather than sending data to a remote server, applications must be able to write data using its local database in the first instance, and then listen for changes from other devices, and recombine them as needed. Apps should utilise local transports such as Bluetooth Low Energy (BLE) and Peer-to-Peer WiFi (P2P Wi-Fi) to communicate data changes in the event that the internet, local server, or the cloud is not available.
  • Devices are capable of creating real-time mesh networks. Nearby devices should be able to discover, communicate, and maintain constant connections with devices in areas of limited or no connectivity.
  • Seamlessly transition from online to offline (and vice versa). Combining local sync with mesh networking means that devices in the same mesh are constantly updating a local version of the database and opportunistically syncing those changes with the cloud when it is available.
  • Partitioned between large peer and small peer mesh networks to not overwhelm smaller networks if they try to sync every piece of data. In order to do this, smaller networks will only sync the data that it requests, so developers have complete control over bandwidth usage and storage. This is vital when connectivity is erratic or critical data needs prioritising. Whereas, the larger networks sync as much data as they can, which is when there is full access to cloud-based systems.
  • Ad-hoc to enable devices to join and leave the mesh when they need to. This also means that there can be no central server other devices are relying on.
  • Compatible with all data at any time. All devices should account for incoming data with different schemas. In this way, if a device is offline and running an outdated app version, for example, it still must be able to read new data and sync.

Peer-to-peer sync and mesh networking in practice

Let us take a look at a point-of-sale application in the fast-paced environment of a quick-service restaurant. When an order is taken at a kiosk or counter, that data must travel hundreds of miles to a data centre to arrive at a device four metres away in the kitchen. This is an inefficient process and can slow down or even halt operations, especially if there is an internet outage or any issues with the cloud.

A major fast-food restaurant in the US has already modernised its point of sale system using this new architecture and created one that can move order data between store devices independently of an internet connection. As such, this system is much more resilient in the face of outages, ensuring employees can always deliver best-in-class service, regardless of internet connectivity.

The vast power of cloud-optional computing is showcased in healthcare situations in rural areas in developing countries. By using both peer-to-peer data sync and mesh networking, essential healthcare applications can share critical health information without the Internet or a connection to the cloud. This means that healthcare workers in disconnected environments can now quickly process information and share it with relevant colleagues, empowering faster reaction times that can save lives.

Although the shift from cloud-only to cloud-optional is subtle and will not be obvious to end users, it really is a fundamental paradigm shift. This move provides a number of business opportunities for increasing revenue and efficiencies and helps ensure sustained service for customers.

Continue Reading

Business

When something personal fills an important gap in the market 

by Cécile Mazuet-Eller, founder of NameSwitch

There aren’t many business ideas that go from a personal experience to filling an important gap in the market. However, this is certainly the case for NameSwitch, the UK’s pioneering and only name changing support service launched in 2018. But what inspired its inception and what challenges did it face? Here, Cécile Mazuet-Eller, the founder of the company, in its seventh year, explains.

My entrepreneurial journey is a bit unusual in that it started from my own experience of going through a divorce, which became a pivotal turning point for me not only emotionally, but practically too. I wanted to remove my married name, and I had a visceral reason to do so as I really didn’t want to keep it. Feeling extremely frustrated at still receiving letters and official documents featuring my previous name, I was desperate to change it but like for so many people it became a stop-start, arduous task.

Once I started the process, I realised it was taking up far too much time I didn’t have; being a single mum to two young children and working full-time is no mean feat, so when I embarked on the name changing process I realised it wasn’t going to be easy.  Searching for a solution to help, all I came up with was a service covering the US and Canada, but nothing that worked for the UK, so in the end, I spent a whole year to get everything changed that had to be, which proved long and stressful to say the least.

Nurturing the idea

In the early days I was fortunate enough to be surrounded by positive people who had good contacts, and who saw the viability of my idea. Living in a small community filled with intelligent and well-rounded people, I wasn’t short of encouragement from them and friends, who recognised as well as I did there was a definite gap in the market. Working with a web development team in Serbia which was also recommended, I enlisted additional help from a university student on some research.

I always wanted to run my own business, and there were several reasons why I needed to embark on something new. As the only breadwinner in the house, there were mounting bills while balancing the demands of motherhood and other financial responsibilities. Cash was limited but what little I had was used carefully which I put into the business.

In the early stages, which included the development of the unique technology that underpins the service, I carved pockets of time at night and on weekends to create a strong foundation for the business. Creating something completely from scratch was like a form of healing, which is why it was and remains such a personal project.

Mulling over the idea for at least two years following the original lightbulb moment, the business was registered in 2015, with time needed for building the robust platform in order to  create a viable product. Drawing on my previous experience, I investigated overseas equivalents, financials and marketing intelligence ensuring there was a genuine need for the service in the UK. Fortunately enough I was able to share my plans with my employer at the time, who turned out to be my biggest supporters, becoming my first paying customer who purchased a NameSwitch for his ex-wife, who was getting married to someone else!

With a career in telecommunications and a degree in marketing, I was already used to hard work and having the support and encouragement from my telecoms team was extremely helpful.   

Support and coaching

Coaching was an important element of the start-up process, obtained through a wider network and some financial support from family,  with no other funding or investment being available.

The challenges

Presented with certain obstacles like all businesses are, there was a lot to juggle and at times it felt like too much but I managed to navigate the complexities involved. When Covid hit that was a huge set-back, given that our biggest target market was and still is, newly-weds. With all weddings being banned, it hit NameSwitch hard, but our saving grace were the people who used the time to change their name’s in lockdown, by doing something they previously didn’t have time for. Being 100% employed by the business by this stage, it turned into a year of survival and another big challenge.  

In 2022-2023 we concentrated on growth for NameSwitch, when me and my dedicated team were satisfied with the service, it was time to consider investment into PR, advertising and partnerships to increase brand awareness to reach the revenues that were needed.

In 2022-2024, it was forecast that 285,000 – 415,000 weddings will take place resulting from the pandemic, which has reflected well on the business in recent years. And amidst the trials and tribulations it’s proved to be both exhilarating and exhausting in equal measure.

With hindsight, there are certain things I’d have done differently, such as bringing in a partner early on to put us in a stronger position sooner, and adding more resource  to improve growth, but I know that’s all part of the steep learning curve and something to take with me to projects in the future.

Advice for aspiring entrepreneurs

For anyone contemplating their own entrepreneurial endeavours, I’d recommend to ‘one hundred percent go for it’ – but do not bet the house on it and whatever happens, embrace the journey.

Continue Reading

Business

How relationships with work are changing

by Amrit Sandhar (CEO/ Founder, &Evolve)

Since Chris Argyris’s work in the 1960s into the psychological work contract, the assumption’s remained that it’s based on mutual exchange of beliefs and expectations of what employee and employer can expect from each other, given a contract only works with two parties agreeing to it.

But have we seen a shift in the balance of this contract, where the expectations of employees have really changed? Since the industrial revolution, organisations dictated employees’ working arrangements which focused on driving greater productivity and performance. This reflected the imbalance of power, with employees reliant on their organisations to structure working arrangements to drive the best results.

Employees signed up to this psychological contract, despite it representing an imbalance in favour of the employer. However, the pandemic stressed this equilibrium, which has led to many, reevaluating their relationship with their work.

While the pandemic has had a long-term impact on most, affecting everything from education to mental health, it could also be the cause of an evolution that’s changing people’s relationship with work. While organisations were supported through furlough schemes and government grants, employees took responsibility for keeping businesses going, by changing the way they worked. Employees took an unprecedented situation and found ways of dealing with it and since the first time in many years, employees had and took direct ownership of the success of the organisations they worked for – which changed everything.

We’ve seen a seismic shift in how we think about work since the that time, which goes far beyond submitting requests for flexible working. It shows that we’re at the threshold of realising a more balanced psychological work contract, driven by employees, who have different mutually agreed beliefs and expectations in how employees and employers work together.

Gone are the days when employees are only satisfied with financial reward and a nice manager. Gen Z will soon become the largest generation making up our workforce and while money is important to them (as they’re likely to be poorer than previous generations), many want work to be something that complements their life, and not something that only provides financial reward.

Some have said the generation gap is a myth, and before the pandemic this may have been true. But when a generation has experienced such a paradigm shift it brings a different mindset of beliefs and expectations about how work can and should be carried out.

It’s hard to see how anyone could go back to the previous way of working, which should have always focussed on outputs and outcomes rather than hours worked. Other than manufacturing, where it was easy to measure productivity, organisations have become complacent in measuring output and outcomes, with employees paying the price for this ambiguity.

Organisations utilising employee engagement surveys, listening forums, and employee representative initiatives often launch them with the best of intentions, however, the historical underlying imbalance of power towards employers, has prevented a more equitable relationship from forming, despite these initiatives. The strain some organisations are experiencing with mounting pressure to challenge how work is carried out, whether from expecting remote working to questioning if a four-day week would drive greater productivity, shows the shift taking place to the long-standing equilibrium of the psychological work contract.

Future successful organisations will be those that can attract and retain the best talent, and it’s unlikely that the next generation of employees will be willing to relinquish their courage to challenge how work is done.

Employees will seek a greater understanding of exactly what’s required of their role and expect organisations to clearly define measures, to understand how their value and success will be measured, regardless of when, where, and how they choose to work.

Rather than resisting change organisations should consider how they can shape it, by questioning and finding solutions to measuring outputs and productivity, by looking at how they help employees feel respected and valued, and how they help bring the psychological contract, based on a new set of mutually agreed expectations and beliefs to life.

Continue Reading

Copyright © 2021 Futures Parity.