Business
Web Scraping in 2022 & Beyond

Source: Finance Derivative
Web scraping has been coming into the limelight in recent years due to the rising interest in data. Businesses across the globe have been eyeing automated data collection as a way to enhance their profitability and overall decision making.
We’ve sat down with the Lead of Commercial Product Owners at Oxylabs.io, Nedas Višniauskas, to talk about the future of web scraping. Few people have been as deeply involved with the industry as Nedas, which has allowed him to gain a unique perspective on how it has developed and how it will continue to do so.
What do you think has been the biggest change in web scraping over the last decade? How has Oxylabs participated in these changes?
There have been some interesting changes during the past few years. One of them, I think, has been the proliferation of increasingly sophisticated anti-bot systems. Scraping such websites at scale, in turn, becomes more difficult.
Scraping enthusiasts, of course, have their own answer to these issues, which is to develop dedicated data collection tools. These, while limiting the field of use, can bypass the anti-bot systems and they are constantly being updated for that purpose.
Another important change has been the rising popularity of JavaScript. More and more websites are using it to load critically important data dynamically, which means it’s essentially unreachable without browsers.
Headless ones, therefore, are a necessity. At the same time, that means infrastructure costs are rising as headless browsers take up much more computing power and traffic than simple HTTP requests.
Finally, ethics have been in the limelight. For example, residential proxy providers are looking for ways to inform and reward participants of the network. We ourselves took charge of building the framework for ethical acquisition, which, I believe, has played a part in the fact that there are less shady practices and more clarity among all industry participants.
To answer the second question, Oxylabs have reacted to these changes with the development of Scraper APIs. We created both dedicated and universal scrapers that can acquire publicly available data from nearly any website without issue. Additionally, all of our proxies are ethically sourced, giving our partners the much needed peace of mind when engaging in scraping.
Have you seen or noticed any particular trends in data acquisition or web scraping? Are specific data types becoming popular?
Off the cuff I’d say that the use of ecommerce and delivery data has been booming since the pandemic hit. Businesses want to (legally) spy on competitors and gain access to as much data as possible. Data types like pricing, products or delivery times are important to any competitor.
But these have always been important. Maybe I would say that external data in general has risen in importance. Outside of that, I don’t think there have been any particular trends in data types. There have been, however, changes in the entire supply chain. As I’ve mentioned, businesses only really need the data. Even then, the data is not the key – insights are.
As such, businesses at the tail-end of the chain have proliferated in recent years. Data-as-a-service aggregators, ones that collect information and sell sets of it, have been rising in popularity.
There are also some businesses that provide insights directly. While these are still few and far between, some of them have unique value propositions that I could see as worthwhile. Jungle Scout, for example, is a service that both scrapes external data and has large datasets from internal sources. As such, they can provide insights other businesses can’t.
What do you think are the biggest challenges the industry is facing currently? Are there any innovative solutions to these or other challenges on the horizon?
Bot protection has always been the greatest challenge. Scraping, you see, is a cat-and-mouse game. Websites attempt to implement anti-bot measures, such as the well-known CAPTCHA, while scraping companies attempt to continue evading them to retain access to data.
There have been great strides made in bot protection. TLS (Transport Layer Security) fingerprinting has been one such improvement. Sophisticated websites can use initial network handshakes to match them with headers. As many scraping tools manually modify the headers sent, TLS can often be mismatched, which would be a dead giveaway.
On the other hand, the deck is always slightly stacked in the favor of scraping. Most anti-bot protection features put a dent in the overall user experience. Filling in a CAPTCHA is something that detracts from that frictionless experience of the modern web we’re used to.
Some businesses use these techniques and see no issue. Others, ones highly concerned with delivering the best user experience possible, avoid using CAPTCHAs unless absolutely necessary. It’s always a tradeoff. More bot protection equals, almost always, worse UX, which leads to less revenue. But then less people are scraping your website.
Additionally, new pages with interesting data and content appear all the time. And you don’t start building a website from bot protection. It has to be functional first. So, the process of scraping is a lot easier than it could be for a long time.
Would you say that there are potential benefits in web scraping for academic research or policy-making? If so, why hasn’t the scientific or political community adopted the practice?
Academic research, quantitative in particular, is in large part based on data that doesn’t exist on the internet, yet. There could be studies, however, on internet behavior or something of the like where scraping could be immensely useful. Additionally, I think we’re not seeing such widespread adoption due to the previously mentioned barrier to entry.
Let’s imagine that there’s no previous scraping experience in some particular university. The researcher would have to build everything from the ground up, get all the deep knowledge, and the funding required just to start acquiring the data.
It doesn’t help that the research areas that benefit the most from scraping (like sociology, economics, psychology, etc.) are far removed from the coding, development, and IT in general. I think it’s more of an unfortunate, but temporary, circumstance, because web scraping providers will be able to reduce the barrier by a significant margin in the future.
When it comes to policy-making, I’m not so sure. I think that rather than making, it should be about enforcing. Governments are definitely knee-deep in web scraping for all kinds of security purposes. Businesses, on the other hand, have been using the same processes to protect themselves from counterfeits and copyright infringement. There’s an entire business vertical dedicated explicitly to brand protection.
You may like
Business
How to identify the signs that your IT department need restructuring

Source: Finance Derivative
Eric Lefebvre, Chief Technology Officer at Sovos
For firms to execute transformations and meet their overall vision, it is crucial that their CIOs are able to recognise the signs that their department is in need of some internal change. In the current economic climate, CIOs working to fulfil their organisation’s priorities and meet business goals might hesitate to acknowledge that their IT department needs restructuring, never mind be able to identify the signs.
However, these problems rarely fix themselves and organisational restructuring requires conviction and determination from leadership for it to occur successfully. So, what are some of the key signs that CIOs should look out for?

Struggling to keep up with industry demands
CIOs unsurprisingly are working in an extremely demanding environment at the moment. Meeting these evolving demands is crucial for companies. When demands are not met and not handled properly, this can have a lasting impact on organisational goals and objectives, and even impact the way in which transformations are put into effect.
Depending on the organisation’s structure, the way in which being unable to keep up with demands manifests itself can differ. Despite double digit reductions across the industry, the search for talent across the tech world continues, project costs continue to rise as the cost of labour has increased and schedules have been disrupted by significant attrition. Many companies will also find business costs, such as that of third-party software, are higher than planned and technology debt continues to pile up faster than it can be sunset.
Whilst leadership teams might dedicate their department’s attention on the factors discussed above, they may find that their team will fall short when it comes to timely deliverables and helping maintain your organisation’s tech stack and guide its business transformations. Looking beyond the immediate problems of high costs and considering an internal reshuffle may be the solution for many IT departments.
Internal conflict within the team
Organisational designs with underlying issues can cause constant friction, especially when they go unacknowledged. An IT department that lives in conflict will certainly be reflected in results and less than successful tech transformations. CIOs will find that by adopting an organisational design which works through staffing issues, will better innovate, especially if they can all work together.
Department leads should have a strong understanding of their team’s work environment and guide them through any long-term or potential problems. When an individual is working in a demanding or complex industry, working well with your team shouldn’t be the main impediment to innovation. By acting quickly to eliminate internal conflict, CIOs can better lead and ensure their team’s focus is entirely on producing more optimal outcomes.
Delays are commonplace
When a large amount of your team’s time is spent setting objectives, budgets and timelines for the projects they are working on, it is vital that they are met. When delays are coming from the IT department, they will inevitably hinder the development of any business transformation, especially if it prompts teams to spend excessive amounts of time rearranging budgets and timelines and therefore hindering innovation.
IT departments are a crucial aspect in many different parts of a company’s transformations, so remaining on track when it comes to timelines and innovation is critical to operational plans. If delays have become commonplace in an IT team, and external factors are impacting projects, CIOs should look at restructuring an IT department to solve these issues.
The strongest team relationships do not happen by accident and are the result of good planning, strong leadership and a motivated team. CIOs can ensure this by providing vision and long-term strategy with clear goals and objectives to produce high levels of quality output.
When internal issues are noticed in an IT department, and are noticeably impacting team morale or productivity, this should indicate the need for departmental restructuring. Be that due to an inability to meet market demands, issues with productivity and meeting deadlines or internal conflict, these issues all risk a department’s functionality and an organisation’s ability to achieve its goals. In short, don’t overlook the warning signs!
Business
Why the future is phygital

Source: Finance Derivative
By Eric Megret-Dorne, Head of Card Issuance Services and Service Operations at Giesecke + Devrient
Digital banking has become increasingly ingrained in people’s everyday lives. Today, 73% of people globally use online banking at least once a month. Traditional bricks-and-mortar banks, which have long relied on the in-person experience with customers, are now having to step up their offering. With new ways of working blurring the work-home boundary, banks must ensure a fast, seamless connection between face-to-face processes and virtual customer experiences.
However, this does not mean that physical and digital banking are in competition with each other. In fact, many continue to use physical bank cards, with 1.12 billion in circulation in 2021, which provides the basis for digital payments and offerings. As a result, the benefits of digitalisation should converge with the comfort of physical touchpoints to create a holistic, “phygital” experience.
The path to phygital
Banks are accelerating their digital transformation strategies to keep up with the fast pace of fintech innovations. To meet the changing needs and preferences of customers, the payment world is leveraging new technologies to create personalised experiences through a range of different channels.
While the digitalisation of banking has been underway for quite some time – particularly for younger generations – events such as the Covid-19 crisis forced banks and customers of all ages to use digital tools and processes to compensate for branch, office, and call centre closures. With branches worldwide typically operating at reduced capacity due to social distancing requirements, consumers embraced online banking to avoid both the virus and potentially long queues.
However, some consumers still enjoy physical touchpoints, meaning a digital-only approach won’t suit everyone.
Striking a balance
It’s all about options – consumers now want to freely switch between traditional and digital channels without being forced into one. But how can banks achieve this phygital balance? One way is to equip physical channels with digital capabilities, so that online tools can augment the physical experience. For example, personalised bank cards with a bespoke design can be activated digitally, offering customers an extra layer of convenience. Having to wait for a new PIN to arrive in the mail is a common bugbear for consumers, so bringing card activation processes into the digital ecosystem will ensure a more seamless experience.
Greater automation in the card issuance and activation process enables the benefits of digital to be integrated into the physical banking experience without being intrusive. For instance, self-service kiosks empower customers to print their own cards, reducing the time between acquisition and card issuance, while still allowing for in-branch expertise if needed.
The personal touch
Phygital strategies also give banks a range of valuable data insights that can help them better serve their customers. This includes data on purchasing behaviours and habits, which can then be utilised to improve banks’ offerings and unify the physical and digital brand experience. Using omnichannel data helps to build a hyperpersonalisation strategy to provide real-time services.
In this way, digital solutions help banks maximise their user experience. Whenever a consumer interact with a bank, it creates data and behaviours. With fragmented databases, legacy systems and real-time data created by interactions with third-party partners through Application Programming Interfaces (APIs), it is not always easy for banks to streamline this data from different sources. By understanding patterns in that data and behaviours, banks can tailor and personalise unique experiences for each and every user.
Where security meets innovation
With big data opportunities abound, banks should be mindful of their consumers’ security concerns. Customers are now demanding much more transparency when it comes to how information is stored and collected. At the same time, they still desire greater personalisation via digital methods. Therefore, any successful phygital strategy requires a robust digital security to ensure customers have the same peace of mind as when they complete physical transactions.
To close the gap between innovation and security, banks should utilise tokenised infrastructure, which ensures the safe provision of payment credentials and securing of customer payments across all touchpoints. This is particularly important as regulations such as PSD2 and SCA demand strong authentication requirements.
The use of a token greatly enhances the consumer experience. For example, it allows for card details to be automatically updated for subscription services upon the expiry of an existing one, avoiding any service disruption. Multi-factor authentication can also ensure an additional layer of security, as it combines a password with verifiable human biometrics such as fingerprints or facial recognition.
Best of both worlds
Every consumer has unique preferences when it comes to banking. Therefore, banks must evolve by bringing both physical and virtual touchpoints into a ‘phygital’ world. Only a phygital approach can meet the needs of all end users – whether they favour an in-person experience, an online one, or a blend of the two. The holistic data insights, personalisation opportunities, and optimised security ensured at every touchpoint are also critical in building future-ready banks.
Business
51% of Apprenticeships Axed: Alternative Ways To Secure The Future of SMEs

More than half of UK-based SMEs expect to increase their workforce numbers by the conclusion of 2023. However, many industries are experiencing a skills shortage problem, instigated by Brexit and a rise in economic inactivity.
One of the solutions has traditionally been the hiring of appearances. Unfortunately, due to the cost of living crisis, SME apprenticeships are under threat. Financial difficulties led to 51% of apprenticeships being axed in 2022, hindering both the job market and smaller businesses that rely on their talent.
Apprentices are valuable to SMEs for several reasons, addressing skills shortages, and allowing businesses to mould the ideal candidate whilst securing government funding.
Luckily, there are several other ways SMEs can dominate their market, with SME-focused digital marketing agency Add People providing their top tips:
7 Practices All SMEs Should Implement To Succeed:
- Invest In Employees
“Employees are obviously one of the most important elements of a successful business.
By investing in your staff, such as rewarding them for hard work, offering incentives and cultivating a space for them to flourish, you can help your SME succeed. From increased productivity and morale to a more positive workplace that attracts top talent, success often begins here.”
- Create A Strong Digital Presence
“The internet should not be underestimated as a tool for generating business. From allowing individuals to find out information, contact you and even purchase products and services, establishing an online presence is essential. Consumers are also more likely to trust and purchase from a business with a visible, credible online presence, so creating a user-friendly website is more essential than ever.”
- Diversify Revenue Streams
“If the last few years of instability have proven anything, it is that diversifying revenue streams is paramount to mitigating risks. Whether the blockage of the Suez Canal or the mass shipping delays caused by the Covid-19 pandemic, too much reliance on a single product can threaten your business.
Expanding into new products and services means SMEs are resultantly capable of reaching new audiences and new sources of revenue.”
- Collaborate & Form Partnerships
“Small-to-medium-sized enterprises can strongly benefit from collaborating with one another, especially across market sectors. These partnerships can provide your business with access to new resources, to enter new markets and improve your brand image within multiple markets.
Similarly, sharing your knowledge with another market can lead to increased innovation, allowing you to develop and improve both existing products and conceptualise new ones.”
- Use AI & Other Technologies
“AI is one of the most exciting developments of the 21st century and is set to revolutionise all industries. SMEs should be taking advantage of implementing AI into their offering, allowing them to stand out in their relevant markets and retain their competitiveness.
AI can also help to improve the decision-making made by a business due to analytics and insights. These can be particularly useful for any markets that are data-driven, but will ultimately help any business with regard to scalability.”
- Adapt To Industry Trends
“ World markets are continually changing, meaning industries are constantly having to evolve. By keeping on top of these changes, you allow your business to remain competitive and attract new customers.
This flexibility is one of the key tools to secure long-term success for any SME, and will allow you to capitalise on new opportunities for years to come.”
- Seek Feedback
“No business will get it right the first time, and the new and unpredictable changes to the market complicate this. Luckily, by always asking your customers and clientele for ways you can improve your business, you gain valuable insights into your consumer demographic and their needs. Learning from this information will allow you to become one of the most valuable and trusted providers within your industry.”
Peter Marshall, Chief Marketing Officer at Add People, a digital marketing agency specialising in small-to-medium-sized enterprises, had the following to say:
“While apprenticeships are a key feature of many SMEs, they are not vital for their success. One of the main reasons that apprenticeships are so popular is the funding that small employers can gain through their recruitment, allowing these smaller businesses to train staff that work to their standards and ethos. This means they are fully trained for a job role when the apprenticeship concludes.
Instead, businesses should focus on long-term solutions at the heart of operations. Making these changes will ensure a healthy future in any market, protecting both the business and the future workforce – including any apprentices!”
Simon Bell, Founder and Director at Careermap, the UK’s leading Early Career website also had the following to add about apprenticeships:
“’Apprenticeships are a win-win situation. Not only for the apprentice but for businesses alike. Training your workforce of the future is vital to keep businesses growing, helping to bridge the skills gap and offering unique perspectives. Reverse mentoring is a hot topic; apprentices can help your organisation do just that by re-energising current employees, encouraging creativity, open-mindedness and innovation.’

How to identify the signs that your IT department need restructuring

The Perfect Sound for a Good Night’s Sleep

Why the future is phygital

The Sustainability Carrot Could be More Powerful Than the Stick!

Hybrid cloud adoption: why vendors are making the switch in 2022 and why you should too
