The Results: Job Title Prediction Outcomes – Deep Learning & Job Descriptions (Pt 3 of 5)

Predicting a Job Title from a Job Description

The obvious first results to look at are what the model was trained to do: predicting a job title for a job description. This is a job description for an infrastructure engineer that was not part of the training set:

 

The job title is included in this job description and is in fact mentioned three times. There are also a lot of detailed duties and skills included for the position. Let’s create a second job description by replacing the job title “infrastructure engineer” with “person” and removing the two other references:

 

Now we run the job title prediction model on both job descriptions and compare the resulting embeddings with the learned job title embeddings from the model using the cosine similarity. These are the 10 most likely job titles under both scenarios:

Original job description Without “infrastructure engineer”
1 infrastructure engineer linux engineer
2 linux engineer devops engineer
3 systems engineer linux systems engineer
4 windows engineer systems engineer
5 devops engineer systems administrator
6 automation engineer devops
7 linux infrastructure engineer senior devops engineer
8 automation engineer devops linux systems administrator
9 devops infrastructure engineer devops consultant
10 linux systems engineer build automation engineer

For the original vacancy the best matching job title was “infrastructure engineer”, which happens to be the correct one. However, most other job titles in the top 10 make sense, although the “windows engineer” is a bit of an outlier.

For the modified job description, the best matching job title is “linux engineer”. It looks like the other 9 job titles would also be acceptable for this scenario. This example shows that the model was able to extract the job title, but it also came up with a meaningful prediction when we removed the actual job title from the job description. This suggests that the model was using duties, skills and other types of information in the description to come up with a prediction when the more direct information was missing.

This is particularly interesting because this was learnt largely from scratch as the model was fed with essentially raw words. All we did was tokenizing the job description into words, lowercasing and that’s it.

Another fact worth mentioning is that the vacancy above had the job title “engineering manager/ project engineer” associated with it. While the job description clearly mentions that the employer is looking for an “infrastructure engineer”, the correct job title was “engineering manager/ project engineer”. As with any real-world scenario, there is a certain amount of noise in the data. Despite this fact, we were quite satisfied with the results we obtained.

The practical applications of this are many, some of which will provide user assistance in platform soon. In simple terms however, it provides us with a mechanism to detect when a job title is probably not an ideal label for a particular vacancy and what a more suitable or common is.

Cleaning Job Titles or User Input

While the model was trained on job description data, we wanted to test it on data that is structured differently: simple queries and raw job titles. These cases have in common that they usually include some irrelevant words and often synonym mappings apply. Some of the examples we tried and resulting job title:

Input Predicted job title
Babysitter St Martins Primary School babysitter
Part-time Library Assistant library assistant
Administrative Assistant – Health Services administrative assistant
Merchandiser (Personal Car) merchandiser
Sr HR Administrator human resources administrator
Call Center Representative I, Full Time, Oscoda, Michigan call center representative
2016-17 Administrative Assistant III – Quail Run ES administrative assistant
MAIL HANDLER ASSISTANT mail clerk
we are looking for a business administrative apprentice and a business administration apprentice

These test data are quite different from what the model was trained on, but it looks like applying the model to these shorter queries results in predictions that are quite acceptable. We found that under these circumstances the cosine similarity that is being calculated for each reference job title has an important role to play. It provides a good measure of confidence for each prediction.

Removing the Obvious

Another type of job title prediction applies to situations where the actual job title is not directly mentioned in a short piece of input text. For humans it is usually clear what the job title should be, but can a model trained on a large number of job descriptions come up with a meaningful answer too?

This table shows some of the job titles that it came up with for various input:

Input Predicted job title
operative for groundworks groundworker
babysitting St Martins Primary School babysitter
we are Pemberton Truck Co and we are hiring regional drivers and a truck driver
taking orders, extensive product knowledge, communication skills, dining skills wait staff
our family is looking for someone to take care of our youngest child and a nanny
we are urgently looking for a developer with a c++ background with a c++ developer
Isuzu Technician (AK4679) auto technician
we are looking for someone to manage a team of five software developers and a software development manager
we are looking for someone to manage a team of five accountants and a finance manager
we are looking for someone to manage a team of five sales representatives and a sales manager

These examples suggest that the model (to some extent) captures information about what someone with a particular job title usually does. For example, it predicts that someone who manages a team of accountants is probably a finance manager. By simply replacing “accountants” with “software developers” or “sales representatives” we jump to “software development manager” and “sales manager”, respectively. The model learned that a person who takes care of a child is often a nanny. When a truck company is looking for drivers, it is probably looking for truck drivers. Given the fact that “Isuzu technician” was not part of the job titles in our training data set, the prediction “auto technician” makes sense.

Finding Related Job Titles

A byproduct of the architecture of our model is that representations are obtained for all job titles in our training set. A job title is basically a short textual label for the responsibilities, duties, experience, etc… that come with a particular job. However, as the “infrastructure engineer” example showed, different people might assign different job titles to essentially the same job (or the same job title to different jobs).

While the CNN in our model generates a representation for a specific job description, we train the model to map those representations close to the representation of the corresponding job title. The representations in our model are embedding vectors that can easily be compared. That is how we did the predictions in the previous sections.

Since we use a large pool of job descriptions and learn to map those to the assigned job titles, we expect to obtain job title embeddings that provide us with a good idea of what a job usually represents.

In order to get a feeling for the quality of the job title embeddings we had a look at the closest neighbours for a particular job title in the embedding space. The table below shows the closest 7 job titles for a “statistician”, “humanties teacher” and “talent acquisition manager”:

statistician humanties teacher talent acquisition manager
statistical analyst humanities teacher talent acquisition specialist
senior statistician qualified humanities teacher talent manager
biostatistician religious studies teacher talent acquisition partner
epidemiologist qts nqt geography teacher talent aquisition manager
research statistician geography teacher secondary talent acquisition consultant
statistical programmer teacher of humanities head of talent acquisition
data scientist qualified history teacher recruitment manager

For “statistician” we find different variations as “statistical analyst”, “senior statistician” and “statistical programmer”. Also some related (sub)fields are detected: biostatistics and epidemiology. Some time ago there was quite a lot discussion going on in the statistical community (e.g. here and here) about the identity of statistics in data science and whether statistics is the same as data science. The appearance of “data scientist” in this list likely means that they share a significant amount of duties and skills in practice.

The spelling error in “humanties teacher” does not prevent the model from coming up with different variations and a list of different kinds of humanities teachers. Despite the fact that our approach does not take the actual characters in words into account, it seems that the model in combination with a large amount of data is tolerant to spelling errors (or at least to some extent). The same thing is observed for the “talent acquisition manager” example where we have a ”talent aquisition manager” in the nearest neighbours.

Some other examples show that relations between “AWS” – “cloud”, “solicitor” – “lawyer”, “property” – “real estate” are recognized:

AWS architect PA to general manager property solicitor
aws consultant personal assistant to general manager property lawyer
aws solutions architect pa to the general manager senior property solicitor
aws developer pa to senior management team residential property solicitor
aws systems administrator personal assistant to the general manager property paralegal
aws devops pa to senior management conveyancing solicitor
devops architect personal assistant office manager commercial property solicitor
cloud architect personal assistant pa real estate solicitor

While there are two job titles in “PA to general manager” we only see personal assistants and no general managers in the closest neighbours. As mentioned, we did not use the characters or words in the job titles, so this is likely because it was learned that those two professions have very different duties.

Continue Reading

This article is part of a series on Using Deep Learning to Extract Knowledge from Job Descriptions. For more information, head to Using Deep Learning to Extract Knowledge from Job Descriptions.

Jan Luts is a senior data scientist at Search Party. He received a Master of Information Sciences from Universiteit Hasselt, Belgium, in 2003. He also received Master degrees in Bioinformatics and Statistics from Katholieke Universiteit Leuven, Belgium, in 2004 and 2005, respectively. After obtaining his PhD at the Department of Electrical Engineering (ESAT) of Katholieke Universiteit Leuven in 2010, he worked in postdoctoral research for a further two years at the institution. In 2012 Jan moved to Australia where he worked as a postdoctoral researcher in Statistics at the School of Mathematical Sciences in the University of Technology, Sydney. In 2013 he moved into the private sector as Data Scientist at Search Party.

Leave a Reply