Setting Up Fill-Mask on Linux

Follow these steps to set up a Fill-Mask system using TensorFlow and Hugging Face's transformers library on Ubuntu.

1. Install System Prerequisites

First, ensure your system is up to date and has essential development tools installed. Open a terminal and run the following commands:

sudo apt update
sudo apt upgrade
sudo apt install python3 python3-pip python3-venv build-essential
    

This installs Python 3, Pip (Python package manager), and necessary build tools on your Ubuntu system.

2. Set Up a Virtual Environment (Optional)

It’s a good practice to create a Python virtual environment to manage dependencies in isolation. Run the following commands:

python3 -m venv fill-mask-env
source fill-mask-env/bin/activate
    

This creates and activates a new virtual environment named fill-mask-env.

3. Install TensorFlow and Hugging Face Transformers

Now, install TensorFlow and the Hugging Face transformers library, which provides easy access to pre-trained models for Fill-Mask tasks:

pip install tensorflow
pip install transformers
    

This installs TensorFlow and Hugging Face's transformers library for text processing tasks.

4. Load a Pre-Trained Fill-Mask Model

To perform the Fill-Mask task, you need a pre-trained model like BERT, RoBERTa, or DistilBERT. Below is an example using the BERT model from Hugging Face:

from transformers import TFBertForMaskedLM, BertTokenizer
import tensorflow as tf

# Load pre-trained BERT model and tokenizer for Fill-Mask
model_name = 'bert-base-uncased'
model = TFBertForMaskedLM.from_pretrained(model_name)
tokenizer = BertTokenizer.from_pretrained(model_name)

# Sample text with a masked token
text = "The capital of France is [MASK]."
inputs = tokenizer(text, return_tensors='tf')

# Get model predictions
outputs = model(**inputs)
logits = outputs.logits

# Decode the prediction for the masked word
mask_token_index = tf.where(inputs['input_ids'] == tokenizer.mask_token_id)[0][1]
mask_token_logits = logits[0, mask_token_index, :]
top_5_tokens = tf.math.top_k(mask_token_logits, 5).indices.numpy()

# Convert token IDs to words and print them
for token in top_5_tokens:
    word = tokenizer.decode([token])
    print(f"Predicted word: {word}")

    

In this example, we use the BERT model. The sentence "The capital of France is [MASK]." contains a masked token, represented by [MASK], which the model will predict. The top 5 predictions for the masked token are then printed.

5. Run Fill-Mask on a Custom Input

You can replace the sample sentence with any text containing the [MASK] token. The model will predict what should fill in the blank. Here's an example:

# Custom text
custom_text = "The largest mammal on Earth is the [MASK]."
custom_inputs = tokenizer(custom_text, return_tensors='tf')

# Get predictions
custom_outputs = model(**custom_inputs)
custom_logits = custom_outputs.logits

# Find the masked token and predict the top words
custom_mask_token_index = tf.where(custom_inputs['input_ids'] == tokenizer.mask_token_id)[0][1]
custom_mask_token_logits = custom_logits[0, custom_mask_token_index, :]
custom_top_5_tokens = tf.math.top_k(custom_mask_token_logits, 5).indices.numpy()

# Print the top 5 predicted words
for token in custom_top_5_tokens:
    word = tokenizer.decode([token])
    print(f"Predicted word for custom input: {word}")

    

This script takes a custom input "The largest mammal on Earth is the [MASK]." and predicts the missing word. You can use any sentence with a masked token.

6. (Optional) Use GPU for Faster Inference

If you have a compatible GPU, TensorFlow can automatically use it for faster inference. To ensure TensorFlow is using your GPU, install the following dependencies:

pip install tensorflow-gpu
    

Then verify that TensorFlow is using your GPU by running the following code:

import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
    

If a GPU is detected, TensorFlow will automatically utilize it for computations.

7. Conclusion

You have now successfully set up a Fill-Mask model on Ubuntu using TensorFlow. You can use any pre-trained model from Hugging Face to predict masked words in a given text. This setup is scalable for use with other models and custom datasets. For faster inference, enable GPU support if available.