yang-zhang/lightning-language-modeling Language Modeling Example with Pytorch Lightning and 🤗 Huggingface Transformers. Language modeling fine-tuning adapts…github.com Language Modeling Example with Pytorch Lightning and 🤗 Huggingface Transformers. Language modeling fine-tuning adapts a pre-trained language model to a new domain and benefits downstream tasks such as classification. The script here applies to fine-tuning masked language modeling (MLM) models include ALBERT, BERT, DistilBERT and RoBERTa, on a text dataset…

I’ve converted LaBSE model weights to Pytorch model weight and shared on https://github.com/yang-zhang/labse-pytorch LaBSE is from Language-agnostic BERT Sentence Embedding by Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang of Google AI. Abstract from the paper We adapt multilingual BERT to produce language-agnostic sen- tence embeddings for 109 languages. While English…

Pytorch’s Tensor class has a storage() and a stride() method. They are not very often used directly, but can sometimes be helpful when you need to take a closer look at the underlying data. (I’ll show an example of using them to illustrate the difference between Tensor.expand() and Tensor.repeat() …

This post is some examples for the gradient argument in Pytorch's backward function. The math of backward(gradient) is explained in this tutorialand these threads (thread-1, thread-2), along with some examples. Those were very helpful, but I wish there were more examples on how the numbers in the example correspond to…

Cross-entropy loss is used for classification machine learning models. Often, as the machine learning model is being trained, the average value of this loss is printed on the screen. But it is not always obvious how good the model is doing from the looking at this value. The formula of…

Here are some notes on setting up docker for Kaggle (especially on installing and enabling nbextensions). I had to do this from time to time and wanted to write the steps down for the record. I put it here in case it’s useful for someone else. Why use docker for Kaggle Kaggle is a good…

For several times I confused myself over how and why a dropout layer scales its input. I’m writing down some notes before I forget again. Link to Jupyter notebook: In Pytorch doc it says: Furthermore, the outputs are scaled by a factor of 1/(1-p) during training. …

This post is an abstract of a Jupyter notebook containing a line-by-line example of a multi-task deep learning model, implemented using the fastai v1 library for PyTorch. This model takes in an image of a human face and predicts their gender, race, and age. The notebook wants to show: an…

Simple example of multi-label classification using fastai v1. %reload_ext autoreload %autoreload 2 from fastai import * from fastai.vision import * In [43]: path = untar_data(URLs.PLANET_SAMPLE) path Out[43]: PosixPath('/home/ubuntu/.fastai/data/planet_sample') In [44]: data = ImageDataBunch.from_csv(path, folder='train', sep=' ', suffix='.jpg', ds_tfms=get_transforms(), tfms=imagenet_norm, size=224) img,labels = data.valid_ds[-1] img.show(title=" ".join(np.array(data.classes)[labels.astype(bool)]))