![\"Lead](\"https:\/\/www.harnham.com\/wp-content\/uploads\/2023\/11\/Chris-Hopkins-Headshot--300x300.png\")
<\/p>\nThe big data and data engineering market is growing at a rapid pace. Valued at 44 billion in 2021, the industry\u2019s market size is expected to reach 120 billion USD by 2027 (roughly \u00a388 billion). Are you interested in breaking into this fast-growing space, but not sure how to get started?<\/p>\n
One of our consultants, Lottie Musgrove, sat down with Chris Hopkinson to learn about his journey into data engineering, and how he managed to break into this extremely competitive field.<\/p>\n
Hopkinson has been working in data and analytics for nearly a decade. Currently working as a Lead Data Architect\/Consultant, Hopkinson is starting a new role as Head of Data Engineering and BI at Evri soon. Here\u2019s a summary of Musgrove and Hopkinson\u2019s discussion.<\/p>\n
<\/p>\n
<\/p>\n
<\/p>\n
Question #1:<\/strong> What motivated you to pursue a career in data engineering?<\/p>\n I used to be an accountant, but I didn\u2019t really enjoy the role. So, I started thinking about what I actually enjoyed, and what I wanted to get out of work. And I realised that I was really interested in how we reported information as accountants, how that information flowed around the businesses, and then how we could improve those information systems.<\/p>\n I\u2019d already started to teach myself SQL and was already starting to automate things in my accounting role to help optimise our processes. So, I realised that there was something there, that this was an area that I was really interested in, and also something that could have a real So, I went ahead and did a master\u2019s in what was then called Business Intelligence and Data Mining, and that\u2019s really what brought me into the world of data.<\/p>\n <\/p>\n Question #2:<\/strong> What were some of the challenges that came up for you during your transition from accounting to data?<\/p>\n For me, it was challenging to get the right type of experience in the early days of my data career, and to find a company that was willing to take a chance on me, to allow me to get the hands-on experience I needed to \u201cbreak in\u201d. Eventually, I was fortunate enough to have someone take me under their wing, and allow me to join their boutique consultancy firm, which gave me an opportunity to work on data projects <\/p>\n Question #3: <\/strong>What have been some of the major milestones or turning points in your career?<\/p>\n My first proper role in analytics was a big deal, but I think the advent of massively online courses really opened the flood gates in terms of learning. I remember going online to learn about R, machine learning, and just finding the whole world of online resources and courses quite fascinating. The online resources I found gave me the opportunity to learn about things that I would have absolutely no chance of learning otherwise. Obviously now, there\u2019s a plethora of material available online, but I feel fortunate that I found those online resources when I did, and that I was able to build my knowledge base around that.<\/p>\n Another real milestone for was getting access to tools like Hadoop and the world of big data. Before then, everything was either in a server or a database, whereas Hadoop and big data created an opportunity to start to do things at a much larger scale. This allowed us to look at and analyse things like network data. It was a learning curve but it really enlightening, it really opened up a world of possibilities. And finally, AWS and the cloud was a big milestone that stood out to me.<\/p>\n The cloud allowed me to start thinking in more of a system\/architecture basis, which really changed my idea of how we can build systems, and what we can do to give our organisation the best chance of garnering valuable insights from its data.<\/p>\n <\/p>\n Question #4:<\/strong> What do you do to keep continuing your own development?<\/p>\n I think the challenge now is finding the signal from the noise. There are so many people talking about data, writing blogs, and so much hype around AI, that finding the actual useful information can be difficult. I look for people that are working for companies that are building data products, or people that are leading AI consultancies \u2013 people that can share ideas and trends they\u2019re sending first-hand as industry experts.<\/p>\n <\/p>\n Question #5:<\/strong> Since you started your career, what have been some of the biggest changes in the industry that you\u2019ve noticed?<\/p>\n It\u2019s moving away from a small and narrow stack that most companies use, to a plethora of different tools that are available through the cloud. Now, you need to start thinking in terms of systems more and more. There are also the changes that have come about through the adoption of software engineering principles, and the creation of DevOps and CI\/CD. I think that data engineering as a field is still <\/p>\n Question #6:<\/strong> How do you stay passionate about a field that\u2019s constantly changing?<\/p>\n I really enjoy learning new things, so learning a new programming language is fascinating to me. What I see is that people who are willing to invest in continual learning really stand out in what\u2019s becoming a really complicated landscape. For me, I really need to see my work have an impact on the business. And one of the nice things about engineering is that it\u2019s a practical application of technical knowledge to business So, making sure my work has some impact on the business keeps me focused and grounded on the things that matter. Working with other people that are passionate really helps too. You can get inspired by people that are excited about something, and by people who are good at what they do, and you can play off each other.<\/p>\n <\/p>\n Question #7:<\/strong> How do you see the role of data engineer evolving?<\/p>\n There could be more separation into specialisms because there comes a point where there\u2019s too much knowledge to expect from a single role. <\/p>\n Question #8:<\/strong> Are there any trends that you think data engineers should be focusing on right now?<\/p>\n Different paradigms of storage (S3 vs. database vs. document store) and processing (via Spark, a single machine, or a streaming platform). Getting a good grounding of all those things will help you make good decisions on what you should be using for a particular business problem that you\u2019re trying to solve.<\/p>\n Also, while you don\u2019t need to become a data scientist, gaining an understanding the needs of that domain will help you understand what you need to know about in order to add value to these different things that are happening in the business.<\/p>\n <\/p>\n
\nbusiness impact.<\/p>\n
\nwith big companies and ultimately get my foot in the door.<\/p>\n
\nBecause anybody can use a generative model to summarise something that someone else has written, I think it\u2019s important to look at the credentials of the people that you\u2019re following and make sure they\u2019ve got the kind of experience that means they can talk into the area of experience that they profess to have.<\/p>\n
\nBefore, you could learn a tool like the SQL server stack, and pretty much be set for life. But now, as more companies adopt the cloud, the number of different services available has skyrocketed, which has changed what you have to do (and what you have to know) as a data
\nengineer.<\/p>\n
\non a path of discovery. We\u2019re still trying to figure out how all these different specialisms
\nintersect and work together.<\/p>\n
\nproblems.<\/p>\n
\nWe\u2019re already seeing this already with the separation of analytics engineering and data engineering, so we may see more things that emerge like that.<\/p>\n