Connect with us

Business

Web Scraping in 2022 & Beyond

Source: Finance Derivative

Web scraping has been coming into the limelight in recent years due to the rising interest in data. Businesses across the globe have been eyeing automated data collection as a way to enhance their profitability and overall decision making.

We’ve sat down with the Lead of Commercial Product Owners at Oxylabs.io, Nedas Višniauskas, to talk about the future of web scraping. Few people have been as deeply involved with the industry as Nedas, which has allowed him to gain a unique perspective on how it has developed and how it will continue to do so.

What do you think has been the biggest change in web scraping over the last decade? How has Oxylabs participated in these changes?

There have been some interesting changes during the past few years. One of them, I think, has been the proliferation of increasingly sophisticated anti-bot systems. Scraping such websites at scale, in turn, becomes more difficult.

Scraping enthusiasts, of course, have their own answer to these issues, which is to develop dedicated data collection tools. These, while limiting the field of use, can bypass the anti-bot systems and they are constantly being updated for that purpose.

Another important change has been the rising popularity of JavaScript. More and more websites are using it to load critically important data dynamically, which means it’s essentially unreachable without browsers.

Headless ones, therefore, are a necessity. At the same time, that means infrastructure costs are rising as headless browsers take up much more computing power and traffic than simple HTTP requests.

Finally, ethics have been in the limelight. For example, residential proxy providers are looking for ways to inform and reward participants of the network. We ourselves took charge of building the framework for ethical acquisition, which, I believe, has played a part in the fact that there are less shady practices and more clarity among all industry participants.

To answer the second question, Oxylabs have reacted to these changes with the development of Scraper APIs. We created both dedicated and universal scrapers that can acquire publicly available data from nearly any website without issue. Additionally, all of our proxies are ethically sourced, giving our partners the much needed peace of mind when engaging in scraping.

Have you seen or noticed any particular trends in data acquisition or web scraping? Are specific data types becoming popular?

Off the cuff I’d say that the use of ecommerce and delivery data has been booming since the pandemic hit. Businesses want to (legally) spy on competitors and gain access to as much data as possible. Data types like pricing, products or delivery times are important to any competitor.

But these have always been important. Maybe I would say that external data in general has risen in importance. Outside of that, I don’t think there have been any particular trends in data types. There have been, however, changes in the entire supply chain. As I’ve mentioned, businesses only really need the data. Even then, the data is not the key – insights are.

As such, businesses at the tail-end of the chain have proliferated in recent years. Data-as-a-service aggregators, ones that collect information and sell sets of it, have been rising in popularity.

There are also some businesses that provide insights directly. While these are still few and far between, some of them have unique value propositions that I could see as worthwhile. Jungle Scout, for example, is a service that both scrapes external data and has large datasets from internal sources. As such, they can provide insights other businesses can’t.

What do you think are the biggest challenges the industry is facing currently? Are there any innovative solutions to these or other challenges on the horizon?

Bot protection has always been the greatest challenge. Scraping, you see, is a cat-and-mouse game. Websites attempt to implement anti-bot measures, such as the well-known CAPTCHA, while scraping companies attempt to continue evading them to retain access to data.

There have been great strides made in bot protection. TLS (Transport Layer Security) fingerprinting has been one such improvement. Sophisticated websites can use initial network handshakes to match them with headers. As many scraping tools manually modify the headers sent, TLS can often be mismatched, which would be a dead giveaway.

On the other hand, the deck is always slightly stacked in the favor of scraping. Most anti-bot protection features put a dent in the overall user experience. Filling in a CAPTCHA is something that detracts from that frictionless experience of the modern web we’re used to.

Some businesses use these techniques and see no issue. Others, ones highly concerned with delivering the best user experience possible, avoid using CAPTCHAs unless absolutely necessary. It’s always a tradeoff. More bot protection equals, almost always, worse UX, which leads to less revenue. But then less people are scraping your website.

Additionally, new pages with interesting data and content appear all the time. And you don’t start building a website from bot protection. It has to be functional first. So, the process of scraping is a lot easier than it could be for a long time.

Would you say that there are potential benefits in web scraping for academic research or policy-making? If so, why hasn’t the scientific or political community adopted the practice?

Academic research, quantitative in particular, is in large part based on data that doesn’t exist on the internet, yet. There could be studies, however, on internet behavior or something of the like where scraping could be immensely useful. Additionally, I think we’re not seeing such widespread adoption due to the previously mentioned barrier to entry.

Let’s imagine that there’s no previous scraping experience in some particular university. The researcher would have to build everything from the ground up, get all the deep knowledge, and the funding required just to start acquiring the data.

It doesn’t help that the research areas that benefit the most from scraping (like sociology, economics, psychology, etc.) are far removed from the coding, development, and IT in general. I think it’s more of an unfortunate, but temporary, circumstance, because web scraping providers will be able to reduce the barrier by a significant margin in the future.

When it comes to policy-making, I’m not so sure. I think that rather than making, it should be about enforcing. Governments are definitely knee-deep in web scraping for all kinds of security purposes. Businesses, on the other hand, have been using the same processes to protect themselves from counterfeits and copyright infringement. There’s an entire business vertical dedicated explicitly to brand protection.

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Business

The modern pay experience and the role of financial education

Attributed to Judith Lamb, CHRO at CloudPay

Despite news that salaries are on the rise, and turbulent economic conditions are still being experienced globally,  few businesses can afford to keep pace with the increase in wage rise demands that much of the workforce is seeking. But with skills shortages being felt across sectors, employers are finding they need an alternative solution to help their staff, without breaking the bank. This is why we believe offering financial education and wellness as an employee benefit can play a significant role.

Better financial benefits

There is a glaring gap in education curriculums for financial education, in the UK at least. This lack of focus in syllabuses means that providing a base level of understanding around financial management is often left to the individual or their employers. While businesses may be committed to supporting the wider skills development of their workforce, providing this guidance on what is often considered a life skill is rarely taken into account.

With the uncertain economic climate that staff continue to face, though, providing this support will be valuable to both the workforce and organisation as it will serve as a crucial attraction and retention tool. Household budgets are being stretched and over-inflated salaries won’t be a sustainable option for anyone. Any ability to make money go further will be welcomed by workers struggling to manage their finances.

It also can’t be overlooked that by improving general money management skills amongst workforces, firms can then gain from the knock-on benefit of enabling staff to make more informed decisions on a professional level as well.

A modern pay experience for modern needs

While financial education will certainly act as an advantage for the workforce, it more broadly follows the trend of providing a modern pay experience as a benefit that is growing in popularity. How and when people are paid, how they access their financial information and what control staff have over their payroll has changed significantly in the last year, and will only continue to do so.

Nowadays, people expect a consumer-like experience in more than just their shopping habits. Online banking has put greater control into the hands of individuals to manage their finances at the touch of a screen. This has translated into payroll as well, and flexibility is a key driver of this change.

While flexible working isn’t always an option or a desire for everyone, these type of benefits are. Permanent employees are questioning why they should wait for a rigid monthly pay day for work they’ve already delivered. This sentiment was certainly more widely felt in the peaks of the Cost-of-Living crisis, but is a challenge to the norm that has been picking up pace for some time.

Flexible solutions

That’s why we’ve seen solutions such as Earned Wage Access (EWA) becoming increasingly popular, largely driven by the demands of the workforce than employers themselves. Streamlining processes for firms and individuals has also become a priority in the modern world of work. Time is hugely valuable to everyone and for that reason, no one wants to wait around for the payroll or accounts teams to give them data relating to their own salaries, taxes or other documentation that they need at any given time.

Everyone is becoming more aware of how their information and data is stored and used, and no one wants to wait around to access their own personal documents when applying for a mortgage, for example. They want instant access and full control.

This demand for a modern pay experience that is underpinned by financial education or support from their employer is only going to increase in popularity. For payroll and finance teams, this really is a prime opportunity to showcase the role they can play in changing the employee experience for the better and improve recruitment and staff retention levels. But it takes a commitment to and investment in the right technology to achieve.

Judith Lamb is CHRO at CloudPay, the expert in global pay solutions

Continue Reading

Business

How will regulations effect the open banking sector?

Source: Finance Derivative

Martin Hartley – Group CCO of emagine Consulting 

Comments on the future of the open banking sector and how it will affect the UK market. 

“The UK Open Banking Sector is still primarily driven by regulation. In my view, two of the major current regulations will remain at the forefront moving forward, namely the CMA (Competition and Markets Authority), which mandated the major banks to provide open banking access to authorised third-party providers, and PSD2 (Second Payment Services Directive), which set the standards for secure data sharing. Cybersecurity regulations will only increase in importance, as will Brexit-related changes as any divergence between UK and EU standards could impact open banking.

“Over the upcoming months, increased data sharing through open banking will add crucial pressures to cybersecurity, likely creating a surge in the sector once again.

“I expect ongoing scrutiny and efforts to enhance data protection measures, potentially leading to more stringent cybersecurity regulations being adopted by businesses. I expect to see more partnerships between traditional banks and FinTechs or consultancy firms as they collaborate to enhance cybersecurity or offer innovative services to plug the gap. Conversely, there could be consolidation within the FinTech industry as companies merge to gain market share.

“When it comes to the size of the business and how it is affected, history has shown us that there are certainly positives and negatives of being an SMB when responding to new regulations. On the positive side, they can leverage their agility and they will have a more personal relationship with their customers, potentially leading to a higher level of trust. However, SMBs may face challenges due to their limited budgets and resources. The larger firms will have much larger budgets, allowing them to have more advanced IT systems and IT security, making it easier for them to integrate APIs and develop the necessary infrastructure.

“The benefits of open banking are endless, and the UK Government is showing their forward-thinking mentality in exploring the idea of implementing the technology to streamline wider services. But, much like anything, there are always pros and cons.

“Open banking would simplify payments for public services, making transactions quicker and more convenient for everyone. As it relies on APIs and authentication protocols, open banking would make payments more secure for the public and it would allow access to digital payments for members of the public who have smartphones but possibly no bank accounts. For any digital implementation, it goes without saying that we need to be aware of the risk of cyber attacks and data breaches. These, combined with the exclusion of non-tech savvy individuals, could mean that certain members of the public may not embrace the change, which poses a risk. There is also the additional cost of providing the infrastructure and this will have to be managed carefully to avoid burdening the taxpayer.

“We have already seen digital transformations in areas such as the GOV.UK Pay System and  there are two main indicators of the success of any digital implementation; adoption rates and incidents. There haven’t been any high profile incidents that have hit the headlines in recent times so that to me is a huge positive and provides a level of confidence. It would be interesting to see how many government departments and agencies have adopted GOV.UK Pay for their payment processing needs to understand the system’s usefulness and acceptance within the government. The government must be committed to continuous improvement and to ensure that the system continues to comply with regulations and consciously drives the adoption rate to hit at least 90% of government departments and agencies.

“A favourable regulatory environment will encourage more banks and third-party providers to participate in open banking initiatives, leading to growth in the UK market and positioning the nation as industry leaders.”

Continue Reading

Business

Advancing green mobility for a sustainable future

Accelerating decarbonisation, the transition to SDVs and reshaping urban ecosystems, are helping revolutionise the global automotive industry

By Amit Chadha, CEO & Managing Director, L&T Technology Services

The world is changing. There is an urgent need for a transition toward sustainable practices to combat the threat of climate change. As global temperatures rise and weather patterns evolve, achieving net-zero emissions by 2050 could still help prevent irreversible damage to our planet.

With global carbon emission levels continuing to rise at an accelerated rate, there is a growing momentum toward addressing the scenario on war footing. As the most visible source of emissions, the automotive industry, and, consequently, the future of mobility, is in focus. By helping accelerate decarbonisation, reshape evolving urban ecosystems, and redefine the global automotive industry – we can help reverse the trend and preserve our shared future.

Green mobility has emerged as a major enabler in this direction. Leading stakeholders are becoming increasingly invested in developing a deeper understanding of the multifaceted realm of green mobility and its potential to shape a sustainable future.

Accelerating decarbonisation: A global mandate

Decarbonising the transportation sector is crucial to mitigate the harmful effects of climate change. Fossil fuel-based vehicles are responsible for a substantial portion of carbon dioxide emissions, exacerbating the greenhouse effect. To accelerate decarbonisation, governments and businesses today need to prioritise the adoption of clean, renewable energy sources, such as electricity and hydrogen, for powering vehicles and other modes of public transportation.

Automakers, recovering from the impact of the pandemic and global supply chain disruptions, are therefore exploring new avenues to meet the rising demand for electric mobility. Electric vehicles (EVs), by eliminating the need for fossil fuel-powered engines, play a vital role in improving overall air quality and have emerged as a promising solution for reducing carbon emission levels. They are capable of meeting the diverse needs of all kinds of drivers and offer affordable mobility and maintenance options. Recent advancements in battery technology, including the growing availability of charging infrastructure and incentives for adoption, have led to a significant rise in the EVs popularity.

However, to achieve widespread adoption of electric vehicles, there is a need to address key issues such as battery disposal, supply chain sustainability, and equitable access to EV technology.

Reshaping urban ecosystems: Driving the frontiers of change

Urban areas are central to the momentum around green mobility transformation. As growing global populations gravitate towards cities – congestion, pollution, and limited availability of green spaces have emerged as major challenges. As a result, cities must increasingly reinvent themselves to promote sustainable mobility and improve the quality of life for their residents.

Smart technologies and vertical green systems can contribute to a reduction in the energy demands of buildings by providing shade and insulation, mitigating urban heat islands, and cooling down public spaces. They also enable carbon sequestration, a reduction in pollution levels, and improvements in biodiversity.

Implementing efficient transportation systems, such as buses and trains powered by clean energy, can further reduce individual vehicle usage, traffic congestion, and emissions. Pedestrian-friendly infrastructures, cycling lanes, and micro-mobility solutions like e-scooters and bike-sharing programs can further help promote eco-friendly transportation choices. At a macro-infra level, smart city technologies and data-driven urban planning practices are helping optimise traffic flow, reduce idling times, and minimise fuel consumption.

Integrating green mobility into urban ecosystems is therefore a win-win proposition – fostering cleaner air, enhanced mobility options, and healthier communities.

From a public health perspective, improved air quality can drive a decline in respiratory and cardiovascular diseases linked to air pollution. Healthier citizens translate to a more productive workforce and reduced healthcare costs, further strengthening the growing impetus for vehicle electrification. The shift towards vehicle electrification offers significant economic benefits, including greater job creation, enhanced research and development, and greater investments in sustainable innovations. A consequent reduction in the demand for fossil fuels, scarce in terms of availability and mostly imported, in turn, helps enhance energy security and stabilise fuel prices.

Software Defined Vehicles: Pioneering the change

The global automotive industry is at the core of driving the emerging frontiers of green mobility. Traditional automakers and new entrants are racing to produce eco-friendly vehicles, and this competitive spirit, in turn, is transforming the industry landscape.

Automakers worldwide need to embrace sustainable practices by reducing their carbon footprint during the production process and implementing circular economy principles. Moreover, investing in research and development of alternative materials and manufacturing processes can lead to lighter, more energy-efficient vehicles. The rise of autonomous vehicles presents an opportunity to optimise transportation networks, enhance traffic flow, and reduce accidents. Leveraging this technology, in combination with electric and shared mobility solutions, can lead to a more sustainable and efficient future for transportation.

Software would play a key role in this direction, delivering a streamlined passenger and driver experience paradigm while ensuring conformity with the evolving regulatory standards. With Software Defined Vehicles (SDVs) increasingly constituting a focus area for major automakers worldwide, the future would witness a greater demand for digital engineering services to unlock new value streams.

The importance of ecosystem partnerships

Automotive industry stakeholders are already working with ER&D partners who can deliver across the value chain and understand each of the key parameters in the EV/SDV ecosystem. However, approaching separate vendors for product conceptualisation, design and development, testing, maintenance, manufacturing and after-sales support can increase costs and complexities.

An ER&D partner, equipped with multi-industry expertise, digital engineering capabilities, and a co-innovation commitment, can help drive transformation initiatives for transportation enterprises, overcoming technology constraints with cross-vertical learnings. Leveraging global delivery capabilities, the partner can also provide computing models that consume less energy, boost performance, and optimise data-led algorithms. In addition, they can enable scalable software stacks that leverage sensors and physical components to provide the safety and performance that electric vehicles need.

ER&D companies are also increasingly being called upon to help redefine focus areas with software, ensuring third-party integration, driving feature deployment, enabling CloudOps and fast over-the-air updates. The rising complexities within the connected car landscape further call for adopting software-defined designs that can overcome multi-layered challenges – ranging from development to subsequent deployment, maintenance, and updates.

A multi-stakeholder approach

Achieving the goal of green mobility demands collaboration among various stakeholders. Governments play a crucial role in enacting policies and regulations that incentivise the adoption of sustainable practices and technologies. Subsidies for EVs, emission standards, and urban planning regulations are some of the ways governments can drive the transition towards greener mobility.

Private sector involvement is equally critical. Corporate sustainability initiatives, investment in research and development, and partnerships for innovative mobility solutions can accelerate the transformation. Additionally, consumer awareness and support for eco-friendly practices are essential in shaping market demands and influencing business decisions.

Advancing green mobility is a pivotal step towards a sustainable future. By accelerating decarbonisation, embracing the transition to SDvs, reshaping urban ecosystems, and revolutionsing the automotive industry, this can combat climate change on a significant battleground. The collective efforts of governments, industries, and individuals are crucial in driving this transformation.

Embracing green mobility is therefore not just about reducing emissions, but rather, about fostering a healthier, cleaner, and more resilient world. It is about our common future –striving together toward a prosperous, inclusive, and sustainable tomorrow.

Continue Reading

Copyright © 2021 Futures Parity.