About me :-
I am from Chenapalli, it is a beautiful village located ~70km from Kanyakumari. I came to north India three years back for my carrier. Presently (8-April-2016) i am in Gurgaon (~2800 km distance). Once or twice a year i go to my home. Every time i go, i bring something to my family individuals. So far i have not given anything to my lovely mom. So this time (2016 Easter holidays) i decided to surprise her with an android phone, hence I bought a Moto G3.
Planning to convince my mom to use Moto G3 :-
I know if i give this phone to her, she is not going to accept it. Even if she accepts, she is not going to use it. Last year we gave a button phone she is not using it now. She does not feed it, does not make any call, rarely accepts any call, does not check call history, sms, etc. When i asked her why are you not using it, she answered, i don’t have time to do all this. I know life in village is quite busy, whole day is not just enough to take care of the family, farms, and animals. I do agree with her, So i decided to convince her by giving this phone with something she likes and she can make use of it while doing other works. She is very pious and so fond of Christian songs. So i bought a 32GB Scandisk memory card(though Moto G3 has 16GB internal) and wanted to fill it with tamil christian songs.
About my songs collections :-
When i was 14 years old my Dad bought me a 100 sony CDs pack, i started collecting Christian songs. In that CD pack Nearly 40 VCD(MPEG) and 20 Audio Disks(MP3, WMA) i have in good condition, I have lots of songs in DVDs(VOB) (Second time Dad bought me DVD pack).
DownloadHelper :-
In 2011 i started using youtube. One day when i was surfing internet i found a christian video song, i liked it and i wanted to download it. I googled how to download youtube videos. I came across DownloadHelper a mozilla Add-on. It was pretty easy use, and helped to download the current playing song and it allowed me to download the desired format (with in the available format).
Movgrab :-
The problem was, if internet went off in the middle, i could not resume the download. So i was looking for alternate options. One day i came across movgrab. It is a c based command line tool. I was using movgrab for sometime till one day i came across a Messianic playlist. I was wondering if i could download the entire playlist.
youtube-dl :-
I googled for downloading a playlist from youtube. I came across youtube-dl. It is a python based command line tool. Installing / updating youtube-dl is very simple. (just one command pip install youtube-dl) Downloading a playlist is even simpler. I am using youtube-dl since 2012, and i am able to download pretty much anything from youtube without any problem. youtube-dl is one of my FAVORITE command line tool now. One day i came across a youtube channel which has 2544 songs, and they update every week. I wanted to download the entire channel. Obviously the first thing i came to my mind was youtube-dl. I started downloading the channel by following command.
Troubles while dealing with huge playlist :-
It started downloading the channel. The problem was my internet. My internet connection was not so good. Average download speed was 70Kb/s. By default youtube-dl downloaded the best available format and i wanted the best format. Average size of each song was 30Mb. I knew it will take weeks may be months to download the entire channel. First day was good, i think it downloaded around 100 songs. I wanted to give some rest to my system. I killed youtube-dl and shutdown my system.
Next day i ran the same command and it parse the playlist again and went through each downloaded songs and if it found any song was partially downloaded, it completed the partial files and went ahead. It took few minutes to start downloading the new song (actually 101st songs). I thought it was ok because it did not download any song for the second time but resumed the partially download files. Second day it downloaded another 100 songs. I thought of giving some rest to my system. I killed youtube-dl and shutdown my system. It was fine till it reached 500 songs. After 500 songs when i ran the command to download the songs again it took a lot of time to reach the new download. It ran through all the indexes and compared with the downloaded songs in the folder.
Simple solution - Don’t let youtube-dl die :-
- I find the process id of youtube-dl by running the following command.
- Suspend the process(PID is 2294) using the following command.
- Put the system in sleep. (May be giving some rest to my system.)
- Later resume the process using following command, youtube-dl will continue from the place where it was suspended.
This was quite helpful while situation such as power cut, wifi/internet failure, home (to/from) work travel. I have used this idea in all such situations and it was helpful. So i thought i have solved one problem. I did not shut down my system for few days. One day my system crashed and restarted automatically. OOPS! the process(2294) died. This was not the first time i encountered Mac crash. I should have foreseen it. So this was not the best solution. I was surfing for another options!
Best solution :-
Yey! found a solution from youtube-dl site. Yeah, it was –download-archive flag. –download-archive FILE - Download only videos not listed in the archive file. Record the IDs of all downloaded videos in it. Well this is what exactly i was looking for. I should have used the flag from the beginning. I was in the middle of downloading a playlist with 2500+ songs and one fourth of the songs were already downloaded. Fortunately i used -t flag which downloaded the songs with it’s original title and youtube id. Next step was extract youtube-id from all the download songs.
I ran the following bash script to extract the youtube-id from all the downloaded files and append them in downloaded.txt file.
This script parsed each file in the current directory and took all the file titles and extracted the 11 char youtube id and appended them with youtube as prefix in downloaded.txt file.
There we go, in no time it wrote all the ids of all downloaded songs in downloaded.txt file. Now all i have to do is just run the following command.
Here downloaded.txt file has all the youtube ids and toDownload.txt has the youtube channel’s URL. In few weeks, i downloaded the complete channel. I follow this channel, if any new upload i run the same command and it downloads the new song. This is how i have been collecting songs from youtube.
youtube-dl in android:-
My post paid mobile plan includes 1GB 3G data. I was looking for an option to download youtube video from my android phone as well. I tried few application from playstore to download youtube songs. I felt no app was as good as youtube-dl. I wanted to use youtube-dl in mobile. I was looking for terminal app android.
Termux:-
Found termux. It is a java based terminal emulator. It works perfectly fine. Installed it on a Marshmallow device. Then installed python and youtube-dl by running the following commands.
Unfortunately it does not work in kitkat(19), it requires at least lollipop(21). My android phone is still running on kitkat.
QPython:-
Then i came across QPython - a python console. QPython has its own shell and allows to run pip command, I installed youtube-dl using pip. Then i tried youtube-dl command, it worked.
My songs collection:-
Currently i have ~400GB songs in Tamil, English, Malayalam, Hebrew(mostly Messanic) and Hindi in my external Hard drive. These songs include .mp3, .mp4, .avi, .vob, .DAT, .wma, .webm, .mkv file formats. I use VLC media player in my system, which plays all the formats without any problem.
My Tamil songs collection:-
I copied only the Tamil Songs(included all formats). It was about 16GB of .mp3 and .wma files and 115GB of different video formats. So have 130 GB of songs. I wanted to fill my mom’s phone with all these songs. My primary target was mp3 songs because she can listen them even if she does some other work. 16 GB was fine i could transfer them to mobile. I felt there could be a lot of duplicate files. (May be same song, same song sung by different artist). I wanted to remove them as well.
Naming conventions and identifying duplicate files:-
When i was going through the songs list i noticed the following. The songs were titled in different formats ( i have collected them from different people, every one has their own naming conventions)
- all small characters,
- ALL CAPITAL CHARACTERS,
- all_small_separated_by_underscore,
- ALL_CAPITAL_SEPARATED_BY_UNDERSCORE,
-
- Song title starts with song number.
- artist name or album name as prefix
- artist name or album name as suffix
- AllCapitalizedWords
- words-separated-by-hyphen
- soMe-title with ouT any pa`ttern.
- some end with a leading space .
I wanted to have the titles as simple as possible and uniform, i preferred to have all the titles in small character separated by space. Some of the folders have the songs title as track1.mp3, track2.mp3, etc. Mostly the audio cd’s which i converted to mp3 using audiograber. I used audiograber while i was using windows. Now i keep audio files as it is(.wma).
I copied all the files except the track files in one folder. Since i was going to use for loop i took a backup of all mp3, i didn’t want to be in trouble by wrong execution of any command.
Solving problems 1, 2, 4, 8, 10:-
First i tried to convert all file names to lower case.
Solving problem 5:-
Trying to remove the numbers from the song title.
Either of these worked but there was a problem. It removed the trailing 3 from .mp3, so now all the songs end with .mp Oops now i needed to append 3 with each file.
Solving problems 5, 6:-
Trying to remove the leading space, this one occurred when i removed the leading number from the title.
Solving problems 3, 4:-
Trying to replace all the underscore with space.
Solving problems 5, 6:-
Trying to remove leading n(10) characters, had to do this when the title was stated with Album name or the artist name
Solving problem 7:-
Trying to remove trailing n(16) characters, Songs which i downloaded from youtube has its ids appended. (11 char + one hyphen + .mp4)
Solving random problems:-
Thats all, renaming part was done for mp3. Applied the same for videos. The audio collections were named as as track1, track2, etc. Old video songs were named ad AVSEQ01.DAT, AVSEQ02.DAT, etc. I had to play the songs and change the titles manually.
At this point of time i was able to identify duplicate files easily just by sorting. And i removed most of the duplicate files.
Converting file formats to save memory:-
Now i have huge collection of songs with proper names. Only problem was memory. I wanted to convert the file formats. There were some video files with a static image(s). Basically they were .mp4 or .mkv files with static images. I wanted to convert them as .mp3 to save some memory. I googled sometime, found ffmpeg and libmp3lame will do the magic. Collected all such files and .wma files in one folder and ran the following script.
It converted all the files in the current directory to .mp3 and this is the end of mp3.
Problems while dealing with video files:-
- I have not installed any extra app to play video in my mom’s Moto G3, i tried to play a .mkv file and the device complained it could not play the song. (assuming it wont play .mov, .vob, .DAT, .webm files)
- I tried to play some hd .mp4 songs, though the video was playing could not hear any audio. I did not happen for all files.
- I always download the best available format (by default youtube-dl downloads best available). Most of the songs come around 100Mb. The mobile device screen size is very small comparing with desktop. I want to convert all the songs to mobile screen size(400x200 - best fit for 5” screen), no mater what actual size(480 or 720 or 1080) is.
To fix all the above problems i just had to run one command.
I know it is going to take a lot of time to complete this. I ran this script one night and slept. Next morning everything was ready. I had to remove the trailing seven characters and append mp4 with each file. (script explained above).
Mission complete:-
After converting the video songs to 400x200, ~100Mb became ~20Mb.
Final output:
Total .mp3 songs - 1543
Total size of .mp3 songs - 8.45GB
Total .mp4 songs - 980
Total size of .mp4 songs - 20.19GB
Then i transfered the songs to Mobile.
A 32 Gb memory card was more than enough to contain all the tamil christian songs i have been collecting for ten+ year!.
file_name_converter.sh file:-
Every time i ran commands in terminal, i added them in file_name_converter.sh file for later reference. At this point the file looked like
At home:-
I reached my home, showed Moto G3 to my Dad and Mom. They told , “yeah it is a nice phone, it has a big display, it looks good”. Then i told my mom, “its for you!”. She was totally surprised and refused. I expected this and i prepared a solution for this. I opened Marshmallow’s file explorer and played one of her favorite tamil hymn. The song was over, no response from her, played another hymn, again no response, played another one, at the end of this song she told, this is a very old song and its very nice (She sings this song in family prayers). Yea, now i have a firm feeling that she will accept it. I opened Music player and played the auto playlist. I enabled shuffle and random options. I kept the phone in kitchen and went to meet a friend. I came after two hours the songs were playing (no wonder, she did not know how to stop it yet). She told me, “most of the songs are awesome”. Then i told her, i can teach you how to play songs in this phone. Once you start a song you don’t have to do anything, you can continue your work, songs will play automatically. You can stop when you want to stop and it is very simple to do. She agreed.
Happiness is helping my parents to use android phone :-
Immediately i taught her to unlock the phone, navigate to apps screen, open music player app and play the songs. After two three attempts playing a mp3 song was a piece of cake for her. Next day i went to my Grandma home, when i came back my mom told me, “I was playing the songs, one song was really really good, so i paused the song at the end, i want to hear the song again”. Hurray! Success!, I wanted to teach her about file explorer or Music player’s view songs options, but i felt it might confuse her. So i told her, “mom i will teach them once you familiar with android”. Then i taught her to play video songs, making a call from contacts, receiving a call. She kept trying all four apps and became very comfortable. After the holidays i came back to work. Everyday when we talk i ask her about the phone and songs. She says she listen the songs, mostly the video songs. I am so happy that she started using android phone.
Thank you guys:-
Thanks to bash, youtube-dl and ffmpeg. Without these tools it would have taken a lot of time to complete. After spending some good time with bash i feel bash is slim, easy to use, each command just does what it suppose to do, provides loads of utility and flags, provides native access to system level informations and configurations. Hey BASH you saved me a lot of time, and i love you, seriously I LOVE YOU!.
Special Thanks:-
At this very moment i thank everyone(some names mentioned bellow) who shared their song collections with me. Some of them shared songs even before i heard about youtube!.
Anila Mary
Anish John
Baby
Baby Sharmila
Bella
Benjamin
Edberg John
Hubert Jeno
Jasmine Sarah
Minu Prince
Ruban
Sudhar
Sylvia Jennifer
Vinoth Raj