Connect with us

Business

Web Scraping in 2022 & Beyond

Source: Finance Derivative

Web scraping has been coming into the limelight in recent years due to the rising interest in data. Businesses across the globe have been eyeing automated data collection as a way to enhance their profitability and overall decision making.

We’ve sat down with the Lead of Commercial Product Owners at Oxylabs.io, Nedas Višniauskas, to talk about the future of web scraping. Few people have been as deeply involved with the industry as Nedas, which has allowed him to gain a unique perspective on how it has developed and how it will continue to do so.

What do you think has been the biggest change in web scraping over the last decade? How has Oxylabs participated in these changes?

There have been some interesting changes during the past few years. One of them, I think, has been the proliferation of increasingly sophisticated anti-bot systems. Scraping such websites at scale, in turn, becomes more difficult.

Scraping enthusiasts, of course, have their own answer to these issues, which is to develop dedicated data collection tools. These, while limiting the field of use, can bypass the anti-bot systems and they are constantly being updated for that purpose.

Another important change has been the rising popularity of JavaScript. More and more websites are using it to load critically important data dynamically, which means it’s essentially unreachable without browsers.

Headless ones, therefore, are a necessity. At the same time, that means infrastructure costs are rising as headless browsers take up much more computing power and traffic than simple HTTP requests.

Finally, ethics have been in the limelight. For example, residential proxy providers are looking for ways to inform and reward participants of the network. We ourselves took charge of building the framework for ethical acquisition, which, I believe, has played a part in the fact that there are less shady practices and more clarity among all industry participants.

To answer the second question, Oxylabs have reacted to these changes with the development of Scraper APIs. We created both dedicated and universal scrapers that can acquire publicly available data from nearly any website without issue. Additionally, all of our proxies are ethically sourced, giving our partners the much needed peace of mind when engaging in scraping.

Have you seen or noticed any particular trends in data acquisition or web scraping? Are specific data types becoming popular?

Off the cuff I’d say that the use of ecommerce and delivery data has been booming since the pandemic hit. Businesses want to (legally) spy on competitors and gain access to as much data as possible. Data types like pricing, products or delivery times are important to any competitor.

But these have always been important. Maybe I would say that external data in general has risen in importance. Outside of that, I don’t think there have been any particular trends in data types. There have been, however, changes in the entire supply chain. As I’ve mentioned, businesses only really need the data. Even then, the data is not the key – insights are.

As such, businesses at the tail-end of the chain have proliferated in recent years. Data-as-a-service aggregators, ones that collect information and sell sets of it, have been rising in popularity.

There are also some businesses that provide insights directly. While these are still few and far between, some of them have unique value propositions that I could see as worthwhile. Jungle Scout, for example, is a service that both scrapes external data and has large datasets from internal sources. As such, they can provide insights other businesses can’t.

What do you think are the biggest challenges the industry is facing currently? Are there any innovative solutions to these or other challenges on the horizon?

Bot protection has always been the greatest challenge. Scraping, you see, is a cat-and-mouse game. Websites attempt to implement anti-bot measures, such as the well-known CAPTCHA, while scraping companies attempt to continue evading them to retain access to data.

There have been great strides made in bot protection. TLS (Transport Layer Security) fingerprinting has been one such improvement. Sophisticated websites can use initial network handshakes to match them with headers. As many scraping tools manually modify the headers sent, TLS can often be mismatched, which would be a dead giveaway.

On the other hand, the deck is always slightly stacked in the favor of scraping. Most anti-bot protection features put a dent in the overall user experience. Filling in a CAPTCHA is something that detracts from that frictionless experience of the modern web we’re used to.

Some businesses use these techniques and see no issue. Others, ones highly concerned with delivering the best user experience possible, avoid using CAPTCHAs unless absolutely necessary. It’s always a tradeoff. More bot protection equals, almost always, worse UX, which leads to less revenue. But then less people are scraping your website.

Additionally, new pages with interesting data and content appear all the time. And you don’t start building a website from bot protection. It has to be functional first. So, the process of scraping is a lot easier than it could be for a long time.

Would you say that there are potential benefits in web scraping for academic research or policy-making? If so, why hasn’t the scientific or political community adopted the practice?

Academic research, quantitative in particular, is in large part based on data that doesn’t exist on the internet, yet. There could be studies, however, on internet behavior or something of the like where scraping could be immensely useful. Additionally, I think we’re not seeing such widespread adoption due to the previously mentioned barrier to entry.

Let’s imagine that there’s no previous scraping experience in some particular university. The researcher would have to build everything from the ground up, get all the deep knowledge, and the funding required just to start acquiring the data.

It doesn’t help that the research areas that benefit the most from scraping (like sociology, economics, psychology, etc.) are far removed from the coding, development, and IT in general. I think it’s more of an unfortunate, but temporary, circumstance, because web scraping providers will be able to reduce the barrier by a significant margin in the future.

When it comes to policy-making, I’m not so sure. I think that rather than making, it should be about enforcing. Governments are definitely knee-deep in web scraping for all kinds of security purposes. Businesses, on the other hand, have been using the same processes to protect themselves from counterfeits and copyright infringement. There’s an entire business vertical dedicated explicitly to brand protection.

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Business

How Agile practices can transform people operations

Bryan Stallings, Chief Evangelist, Lucid Software

Fostering a positive workplace can be more challenging today than in years past. As hybrid work solidifies itself as the standard, HR professionals are tasked with navigating a more complex set of responsibilities and expectations. For instance, employees may feel that having a desirable work-life balance is as important as other company-provided benefits, and they may request additional support to accommodate their expectations. While this adds a layer of complexity to the traditional HR role, it presents an opportunity for HR leaders to improve the current workplace experience and differentiate themselves from competitors in order to attract and retain the best talent.  

With 57% of employers having hard-to-fill vacancies, adopting an approach to HR that offers this flexibility can help create the workplace experience needed to attract the best talent in the coming years. Agile practices could support this necessary shift, helping to support people more effectively in an uncertain and turbulent environment.

Agile – supporting a changing workforce 

Traditional HR departments are often perceived as rigid bureaucracies, reliant on established policies and procedures to preserve the status quo. However, organisations increasingly recognize the need for a more agile approach to HR to meet the demands of a dynamic workforce.

The agile mindset encourages flexibility to create and respond to change, test our ideas, and succeed despite the uncertainty that emerges during an initiative. It recognises that the requirements of any initiative are emergent – rather than defined prior – and so teams can  prioritise working closely with stakeholders throughout and in response to change. 

Taking an agile approach to traditional HR functions also improves performance more broadly. For instance, in performance management, managers provide more consistent and relevant feedback so employees are aware of their strengths and areas of growth in real time. Rather than waiting for annual performance reviews, employees working in an agile environment are cognizant of what they need to improve through ongoing communication and collaboration with colleagues. 

Putting agile into action

As HR professionals learn and instil agile practices in their organisation, the journey encourages them to look beyond traditional hierarchy and management philosophies. However, it’s important to recognize that outside of IT and technology spheres, many employees may be unfamiliar with agile ways of working, which can hinder its initial acceptance.

For instance, when selecting members for a team, prioritising an individual’s skills over their seniority can lead to significant productivity improvements by better aligning project ownership with capabilities. To achieve this, first identify the skills your individual team members have and then visualise that data, grouping employees based on their assigned tasks and competencies. This can reveal information needed to understand who works best together or even where the organisation may lack specific talent.

By sharing this information across an organisation, employees are empowered to self-organise their teams for new initiatives based around what skills are needed rather than who is available. And the issues that frequently plague siloed organisations – poor communication between teams leading to delays – occur far less frequently as teams work cross-functionally in an agile way.

Adopting an agile framework can help by providing a structure to guide employees in agile ways of working. There are frameworks, like Kanban and Scrum, that can help. Scrum structures work into a regular process of sprint planning, two-week or weekly cadence blocks, and concludes with a sprint review. They can provide the necessary scaffolding for colleagues to understand how agile principles can apply in practice and across their other work. 

It is also possible to test the waters before adopting an agile framework across an entire organisation. Whether it’s recruiting talent for very specific roles or measuring employee engagement, the approach encourages collecting actionable data on how initiatives are performing, which helps provide the evidence needed to run successful trials and pilot programs and make informed decisions.

Embracing Agile for HR transformation

Too many companies are tethered to outdated HR models that no longer align with the realities of today’s workplace. Embracing agile provides an opportunity to evolve practices and usher in better HR operations. With its flexibility, collaborative ethos, and emphasis on continuous improvement, agile is the natural solution. Applying agile practices not only empowers HR teams to navigate the challenges of the chaotic work environment, but also serves as a catalyst for streamlining processes, enhancing job satisfaction, and cultivating an adaptive, team-centric culture.

Continue Reading

Business

Dealing with Parental Leave: How Your Business Can Support Employees with Families

Looking after your staff is a fundamental part of running a successful business, ensuring staff turnover remains low and workers remain happily motivated. Workers now have more agency than ever when it comes to choosing their employer, in part thanks to the rise in remote working which means workers are no longer limited to looking for roles within their local area.

39% of UK workers now work at home within a given week and workers are beginning to demand more in terms of employee benefits, especially when it comes to welfare.

One of the areas where employees may focus is “family-friendly” working and benefits. But what does the law say about these contractual offerings? And how can your business benefit from having a comprehensive “family-friendly” benefits package? We spoke to the employment law specialists at Beecham Peacock to discover how your business can become more caring.

What does the law say about parental leave?

In the UK, women are able to take up to 52 weeks of maternity leave. The first 26 weeks of leave, which includes two weeks of compulsory leave (four weeks for factory workers), are known as ordinary maternity leave, while the final 26 weeks are known as additional maternity leave.

During maternity leave, a woman’s rights to pay rises, accrued holiday, and returning to work are protected by the law.

Eligible mothers-to-be are entitled to be paid statutory maternity pay for 39 weeks. This will depend on whether or not they satisfy service and earnings criteria. Otherwise, they may not be able to claim maternity allowance.

Statutory maternity pay equates to six weeks paid at a rate of 90% of average weekly earnings (before tax). For the remaining 33 weeks, the current rate of payment is £172.48 or 90% of their average weekly earnings – whichever is lower. This rate is reviewed annually.  

There has been much discussion about the mandatory amount of maternity pay and whether it does enough to support women in the workplace – a recent study found statutory maternity pay is just 47% of the national living wage. To attract and retain women, businesses may wish to consider offering enhanced maternity pay and benefits packages.

For partners, leave entitlements are different. Statutory paternity and adoption leave entitles fathers/partners to take one or two weeks of paid paternity leave, paid at a rate of £172.48 or 90% of their average weekly earnings – whichever is lower. This rate is also reviewed annually.

When this leave is taken differs depending on whether paternity or adoption leave is being taken. Again, your business may wish to consider enhanced leave and pay packages.

For eligible parents, another option that is increasingly taken up is shared parental leave. Whilst the mother will always have to take two weeks of compulsory leave (four weeks for factory workers), the remaining 50 weeks (or 48 weeks for factory workers) can be taken by either parent.

This gives both parents flexibility and the opportunity to spend time with their child.  Statutory parental leave pay is paid at the same rate as the latter part of statutory maternity or paternity pay, and can be paid for up to 37 weeks to eligible employees. Again, businesses may wish to consider offering enhanced parental leave pay to attract and retain employees.

What are the positives of greater employee benefits for parental leave?

Of course, there are extra costs associated with paying more than the statutory pay requirement. However, offering parental leave options and policies that go above and beyond the minimum requirements can benefit a business just as much as it benefits your employees. Such packages will enable business to attract and retain employees.

How to draft a comprehensive parental leave policy

Lisa Branker, Head of Employment Law at Beecham Peacock, advocates for a comprehensive leave policy that supports all of your employees. She comments:

“Entitlements and eligibility for parental leave, pay and benefits should be clearly contained in your business’ relevant policy. If your goal is to attract and retain your workforce through flexible and/or enhanced benefits packages then this information needs to be clearly set out and accessible. A clear policy makes employees aware of how much leave and pay they are entitled to, helps managers to respond to any queries, and allows your business to plan for and support working parents.”

“Pay and leave aren’t the only considerations – for example, your business may be able to offer a salary sacrifice scheme to make childcare arrangements. Other, non-financial support can also be a huge help for new parents or parents-to-be. Increasing the flexibility of working hours or offering a hybrid working scheme can give your colleagues the support they need to manage the transition into parenthood. These measures will enable you to motivate and retain your workforce, without creating an onerous financial burden.”

Every company is different – and there’s unlikely to be a one-size-fits-all solution. Think about which solution (or combination of solutions) is best-suited to your company before creating or amending a parental leave policy. If you’re considering creating or updating your policy, Beecham Peacock’s free policy reviews are a great starting point to check your offerings meet your business and legal needs.

https://www.beechampeacock.co.uk/employment-law/

Continue Reading

Business

How Africa’s largest payments network is integrating social mission with its business aspirations

Being deliberate about creating a “greater purpose” is essential to building an authentic corporate culture, engaging stakeholders, and navigating the evolving landscape of corporate philanthropy. This is the philosophy behind Africa’s largest digital payments network, Onafriq’s, extensive growth and vision to unify the continent’s digital payments landscape according to its General Counsel and Chief Risk Officer Funmi Dele-Giwa.

Dele-Giwa recently shared insights into the organisation’s unique position at the intersection of social impact and commercial ambition at the Women in Payments Symposium EMEA, held in London. During her speech she delved into the company’s journey in delivering greater financial access and connecting all of Africa into a single integrated network that empowers both individuals and businesses.

“The purpose of Onafriq from the very onset was one of providing financial access to marginalised individuals on the African continent and having a positive impact in the countries we operate in and the clients we serve,” she said. “That is why Onafriq was built on the back of a strong belief that mobile money would serve as a strong enabler of financial access to millions of under- or unserved Africans.”

Established nearly 15 years ago with the mantra of “making borders matter less”, the company aims to facilitate cross-border payment services within Africa – as well as in and out of Africa. This is underpinned by the vision of its Founder and CEO Dare Okoudjou, that making a payment anywhere in the world, to anywhere across the globe should be as easy and as painless as it is to make a phone call.

Today, Onafriq’s payments network connects more than 1,300 cross-border payment corridors providing access to more than 500 million mobile wallets and 200 million bank accounts across 40 African markets. This vast digital infrastructure is a testament to its position as the “network of networks”, enabling services like cross-border payments, remittances, card issuing, agency banking and more, which facilitate seamless money flow from, to, and across the continent.

During her talk at the symposium, Dele-Giwa noted that remittance services were a key example of this marriage of concepts, having particularly emerged as a powerful tool for boosting economic growth and financial empowerment. By partnering with international remittance companies, the Onafriq network enables the significant pool of migrant workers from Africa in the diaspora to send and receive money efficiently and affordably. She notes however, that remittances are not just the privy of the global north to south, as there is significant intra-Africa remittance demand which has traditionally remained unmet. Through partnerships with mobile network operators (MNOs) across the Continent, Onafriq is bridging gaps between countries like Kenya and Uganda, as well as Cameroon and Nigeria, by digitising and facilitating intra-Africa remittance flows.

“Strategic collaborations between key sectors of Africa’s financial services landscape are key to unlocking the full potential of remittances as a catalyst for economic growth and development,”  said Dele-Giwa. “As such, fostering robust partnerships between payment networks and mobile money platforms is important to enabling greater remittance flows given the widespread adoption of mobile wallets across the continent.”

Another way that Onafriq is blending the principles of social betterment with business objectives is by empowering small businesses in Africa to flourish and grow by enabling access to a wider range of choices in disbursing or collecting digital payments over cash. Onafriq’s partnership with One Acre Fund is an example of how the company’s network has contributed to providing small-scale farmers with asset-based financing services.

“Our work to open up markets and connect people to opportunities continues to empower the African gig economy, enabling GDOs to deliver cash assistance to needy communities and international merchants to pay local creators, influencers and artists, as well as helping small traders to sell their goods across borders, by simplifying the ways they can pay and can get paid,” said Dele-Giwa.

Another notable aspect of Onafriq’s journey of positive social impact, according to Dele-Giwa, is its commitment to empowering women. Through its agent network in Nigeria, women entrepreneurs are able to generate additional income by becoming agents, and by using the Baxi point of sale device they can easily manage payments for their shops and market stalls. Furthermore, partnerships with organisations like the One Acre Fund helped to empower women in small-scale farming, amplifying their economic participation.

For those seeking to emulate Onafriq’s success, Dele-Giwa noted that it was important to align their social mission with the innovation and collaboration needed to achieve a positive impact while pursuing commercial success.

“Let’s remember, it’s not just about the services we offer. It’s about the impact we make while doing so,” she said. “It’s important to share those impactful stories of empowerment and positive change delivered as a result of your products and services, but it is also important to create a set of impact metrics to measure success by. This way you are always able to hold yourself accountable to employees, shareholders, regulators, clients, and other stakeholders.”

Continue Reading

Copyright © 2021 Futures Parity.