Compiling Android Source Code Froyo (2.2) on Ubuntu 10.04

A few days back, one of my friends gave me compiled version of Android Froyo (2.2) source code. And without compiling it myself, I just used it as it is for my purpose. And after reading some installation guides on the internet, I thought, I should write a tutorial on it so that me and others can benefit from it.

But it turned out to be a bad decision by putting a tutorial without even testing it. I feel sorry for others too. I tried to compile the same compiled source code again, it began to show me a lot of errors. So without going into what stupid things happened to me,  I would highly recommend you to install Ubuntu 10.04 as this was the only version which worked for me. And the reason is Ubuntu 12.04 and Ubuntu 12.10 had gcc versions which were higher than the required version for compiling Froyo (2.2). And I didn't know if cross-compiling for GCC versions was possible or not, I just installed Ubuntu 10.04.

One more thing, if you have Thinkpad T420 or any latest product, Ubuntu 10.04 was released before these products. So even if you install it successfully, you might not be able to view the screen text due to lack of audio/visual drivers.That's what happened to me, so I ended up installing it on Virtualbox.

Anyway, here is the recipe.
 
Step 1:

Install Ubuntu 10.04 either dual boot or as a virtual machine on Virtualbox. If you are using Virtualbox, reserve 2 GB of memory for the virtual machine. It gave me error for 512 MB.

Step 2:

If you don't want to go into the hassle, I have shared android source code through my GoogleDrive Account here. It's around 2 GB in compressed format.

else Download the Android source code from google source code.. I downloaded source code for Froyo which had size of around 6 GB. Make sure, you have curl installed.

Run following commands:

$ mkdir ~/bin

$ PATH=~/bin:$PATH

$ curl https://dl-ssl.google.com/dl/googlesource/git-repo/repo > ~/bin/repo $ chmod a+x ~/bin/repo

$ mkdir android

$ cd android

$ repo init -u https://android.googlesource.com/platform/manifest -b android-2.2.2_r1

$ repo sync


Step 3:

$ sudo -i

Type your login password and then, you wont have to type 'sudo' on later stages.

Step 4:

$ add-apt-repository "deb http://archive.canonical.com/ lucid partner"
$ apt-get update

Step 5:


Install Java-5.

$ sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu hardy main multiverse"
$ sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu hardy-updates main multiverse"
$ sudo apt-get update
$ sudo apt-get install sun-java5-jdk

Step 6:

Install git with:

$ apt-get install git-core

If you are running OS other than Ubuntu, find your command here.

Step 7:

Install the required dependencies. It's one command. Note that plugin list on AOSP site is not correct.

$ sudo apt-get install git-core gnupg flex bison gperf build-essential \
zip curl zlib1g-dev libc6-dev lib64ncurses5-dev \
x11proto-core-dev libx11-dev lib64readline5-dev lib64z-dev \
libgl1-mesa-dev g++-multilib tofrodos

Step 8:

Run following commands to set path variables and compiling your android source code. Compilation may take a few hours depending on your system specifications.

Assuming you are in /home/mohsin directory and your downloaded and unzipped code is in this directory such that /home/mohsin/android/out is a valid directory.

$ cd android
$ source build/envsetup.sh
$ lunch full-eng
$ make -j4

Setup the environment variables:

$ export ANDROID_PRODUCT_OUT=/home/<userName>/android/out/target/product/generic

$ export PATH=$PATH:/home/<userName>/android/out/host/linux-x86/bin

Restart the terminal.

Step 9:

After completion of step 8, lets check if it was compiled successfully.
Run following command.

$ emulator

If you see something like "No command found 'emulator'. Did you mean Qemu" etc., that does not mean, it was not successful.

I have a temporary fix right now. It will work only on currently working terminal like every time, you open a new terminal, you need to run these commands before running 'emulator' command.

Temp Fix#1:

$ source build/envsetup.sh
$ setpaths

Temp Fix#2:

$ source build/envsetup.sh
$ set_stuff_for_environment

Now run $emulator and it should open an emulator.

If it works, we are done for the part of compiling Android Source Code which have been a pain in the ass for me for a few days.

Finding system calls of Android APKs without rooted device

You can use strace command to find out all the system calls being made from your Android application on emulator if you have source code of the application and running it on Eclipse. I have explained it here. However, if you want to get traces from any application running on your device, you cannot do this by following the steps given in the link. And the reason is that 'strace' command on the device does not have root privileges to execute and so, you will get error 'permission denied' in that case.

There is one work around for this scenario by pushing your APK on Android emulator and then use 'strace' over there. There may be some limitations of emulator which can limit system calls information, but at this stage, I don't know much.

1) Get APK for your application. If you already have installed application which you want to strace, you can get your APK via this method. If you have not installed it and don't want to do either, follow this method or this one, though I would recommend the first one because of easiness.


2) Go to Eclipse => Window => AVD Manager and start any emulator.

3) If you have installed Android SDK with the default settings, you should be able to find 'adb.exe' at "C:\Program Files (x86)\Android\android-sdk\platform-tools" location.

4) Copy your APK file in 'platform-tools' folder on your computer.

5) Go to Start button on your Windows machine, type 'cmd' and you will see command window opened up. Now type 'cd C:\Program Files (x86)\Android\android-sdk\platform-tools' on the command window and you will be directed to platform-tools directory. Type 'ls' to see content of this folder and you can find adb.exe here.

6) Now type 'adb devices' which will show you all the emulators and devices attached with the adb debugger. Since we have just one emulator running in this case, we will find only one emulator information. For example, on my screen, following information is displayed.

List of devices attached
emulator-5554   device

emulator-5554 is emulator ID here. If you have devices attached, you would see their serial numbers here.

7) Type 'adb install <myApp.apk>'. It will take some time and your file will be installed on your emulator.


8) Type 'adb -s emulator-5554 shell' on the command window if you have more than one emulator running otherwise just type 'adb shell'. You will get shell of the emulator or your device. Now if you type 'ls' on the shell, you can see files of your emulator or the current directory of your smart phone.

9) Type 'ps' on the command window to find out which processes are running on your device, and one of them is our uploaded application. It might be hard to find myApp.apk application there, but if you know package name (com.mysite.myapp, for example), you will see the package name in the output of 'ps' command. Just note down is PID. Remember our goal is to find out system calls against our application, so we will use PID there.

10) In my case, I wanted to know trace for system calls in a file for later analysis but my running application was constantly outputting something on the screen, making it harder to analyze on the spot. So I needed some file on the emulator where I could put my data but the problem with the emulator was that I was not able to create a file by simply running 'touch' or 'vim' command. So I used a work around.

Create a file trace.txt on your computer. Go to Eclipse ==>  Windows => Other Perspective => DDMS.

 11) Under the 'File Browser' tab, select 'sdcard' folder, though you can click on any folder.

12) Now upload trace.txt file by clicking on 'Push a file to a device' button.

13) On the command window, move to 'sdcard' or other folder where you have uploaded 'trace.txt' file using 'cd sdcard' command.

Now type 'strace -p <pid> -o trace.txt'. All system calls will be placed in this file when you are playing with the application. When you want to finish stracing your APK, press Ctrl+C when you are on command window to stop this process.

14) You can download this trace.txt file to your computer the way you uploaded this file by clicking on 'Pull a file from the device' button. You can analyze the traces of your APK with this file now.

Happy stracing. =)

Capturing screenshot of an Android device through Eclipse

1) Connect your Android phone to computer through USB cable.

2) Enable debugging option on your phone. I have an old phone, and on this phone, it's at

Settings => Applications => Development => USB debugging. Enable this check box.

3) Open Eclipse. Go to


Window => Open Persepective => DDMS

4) Click on 'Screen Capture' button as shown below.

5) Click on 'Save' button and save it on your computer.

Download/upload your Android phone data through Eclipse

1) Connect your Android phone to computer through USB cable.

2) Enable debugging option on your phone. I have an old phone, and on this phone, it's at

Settings => Applications => Development => USB debugging. Enable this check box.

3) From notification bar, click on 'USB connection' and enable 'Charge Only' option.

If you select 'Memeory Card Access', you will not be able to access content of sdcard.

4) Open Eclipse. Go to


Window => Open Persepective => DDMS

5) Now click on 'File Browser' tab.

6) Click on any file and click on this button which says 'Pull a file from device'.

Downloading multiple Android APKs through GooglePlay-Crawler

To date, this is the best code and technique I have seen to download multiple Android apps from Google Play. You can search apps via category (Cricket etc.) and then download all of them in one go, however, maximum limit for downloading apps in one run is 100. You can change the offset and download remaining other apps.

Requirements:

Java
Maven

I will go installation for Maven and Protobuf one by one as Java was already installed on my computer and generally you can find many guides on internet about it.

1) Install JDK and make sure your environment variables are pointing to JDK folder, not to JRE.

My focus is on JDK folder, you should take care of other settings. To be specific, it should point to

C:\Program Files\Java\jdk1.7.0\bin

instead of

C:\Program Files (x86)\Java\jre6\bin

2) Download latest maven from here. I downloaded maven 3.0.4. Extract it to some folder. 

For example, on my computer, it is at 'E:\apache-maven-3.0.4-bin\'. Now add is path to your system variables. 

Go to Control Panel => Systm => Advanced System Settings => Environment Variables.

Under 'System Variables', find 'Path' (Case-insensitive) and click on it. In 'Variable Value', add this text 'E:\apache-maven-3.0.4-bin\apache-maven-3.0.4\bin'. Remember to have ' ; ' before and after this text.


3) Open command window and type 'mvn -version'. It will show information about your current maven installation. That's it for maven. =)

4) Download Googleplay-Crawler from here.(If you are newbie on Github, find 'ZIP' button on the page and click on it. Extract it.

5)  Import this project into Eclipse. If you have not installed Eclipse, you can compile and run it normally through javac commands on command line.

Before running it, you need google username, password and device id.

Type *#*#8255#*#* on your phone and against 'Device Id', you will find 'android-<device id>'. Only <device id> part is your device ID.

Also, your google username and password should be the same which you used on your device as your device id is associated with it.

Now enter this information in 'TestGooglePlay.java' file. You can test your project by running this file. It downloads only one APK as a test, apart from doing other things. If you are a programmer, you know what to do now. =)

Credit: Full credit to Akdeniz for development of this crawler who credits back to  egirault for development of Google Play API.

Strace your Android application with adb

I have been looking on the internet for two days to figure out how can I get information about system calls being made in an Android application when I tap some button on the screen. I got it working  now, so thought I should post it also so that if somebody needs this information, (s)he can directly jump to the right solution.

I assume, you already have installed Android SDK. You can create a new project in Eclipse, with some activity and buttons on the screen because we need to tap some button on the screen (Its just an easy approach) and see system calls on the screen.

I am using DatePicker module, which comes from the Android source code and I will be testing on emulator, though I am sure it would be working on any Android phone also. From now on, I assume you have working emulator and your project ready in Eclipse. (If you have device, connect it with the USB to your laptop and enable 'USB Debugging' mode on your phone settings).

1) Click on 'Debug' icon in Eclipse. It will start an emulator to run your application. Let it run.

2) Now if you have installed Android SDK with the default settings, you should be able to find 'adb.exe' at "C:\Program Files (x86)\Android\android-sdk\platform-tools" location.

3) Go to Start button on your Windows machine, type 'cmd' and you will see command window opened up. Now type 'cd C:\Program Files (x86)\Android\android-sdk\platform-tools' on the command window and you will be directed to platform-tools directory. Type 'ls' to see content of this folder and you can find adb.exe here.

4) Type 'adb help' on the command window to see if adb is working.

5) Now type 'adb devices' which will show you all the emulators and devices attached with the adb debugger. Since we have just one emulator running in this case, we will find only one emulator information. For example, on my screen, following information is displayed.

List of devices attached
emulator-5554   device

emulator-5554 is emulator ID here. If you have devices attached, you would see their serial numbers here.

6)  Type 'adb -s emulator-5554 shell' on the command window. You will get shell of the emulator or your device. Now if you type 'ls' on the shell, you can see files of your emulator or the current directory of your smart phone.

7) Type 'ps' on the command window to find out which processes are running on your device, and one of them is our DatePicker application. It might be hard to find DatePicker application there, but if you know package name (com.datapicker in my case) in your Eclipse project, you can find out the package name there. All you need to note down is PID. Remember our goal is to find out system calls against our application, so we will use PID there.

8) Type 'strace -p <pid>' on the command window. Replace <pid> with the PID value found in the previous step. It will not show enough information about the system calls by now.

9) Now if you click on any button at your application on emulator, you will see lot of system calls being made on the command window. That's all we wanted.

Thanks.

Get 'adb devices' working to make your device debuggable

1) Connect your device to your computer with USB cable.

2) Go to Settings->Applications->Development and enable USB Debugging.

3) Set USB settings 'Read Only' from the notification bar if you want your device to read data from SD card inside your phone.

4) Open this file 'C:\Users\<user>\.android\adb_usb.ini' and put your device Vendor ID after the commented header.

5) Open CMD and go to  'C:\Program Files (x86)\Android\android-sdk\platform-tools' and run 'adb devices'. It should show your device listed there.

If this does not work, here are a few things you should try:

i) Restart your adb server by following commands

adb kill-server
adb start-server

ii) After making changes to your settings, detach and re-attach your USB cable.

iii) Go to Device manager (My Computer -> Manage). Android Phone should be listed on top. Uninstall its driver and enable this check box 'delete driver software' also.
Install/update your driver as given in this link. Repeat step i and ii if it does not work.

Crawling information about Android Apps through Android Marketplace Crawler

     
Note: On my machine, following crawler was able to crawl only around 50 Apps. So, if you know a little bit of programming, you can use this link to download multiple APKs from Googe Play.

1) Open this link . Click on 'create a clone' , fill out information and you would be able to see your created clone link as mine one.

2) Click on Download zip  (this is my clone code) button on your web page and you will get all the updated source code for Marketplace.  I would recommend you to download from the your own clone as it would be updated code and also, my clone code might be unstable due to changes.

3) Unzip this source code and you will have two folders: crawler and server.

4) Now as per requirements given on Android-Marketplace-Crawler page, we need to install 'ant'. 
Here is the installation instructions for 'ant'.

5) Java:

Since you want to work on Crawler, I assume you already have installed Android SDK and Java on your machine. I am not sure if you really need Android SDK installed on your system for Crawler project. However, I had it on my PC before working on this project.

6) As given on Crawler Home page, you need a valid Google Account to run the crawler. Open and modify the constructor, getters and getUsers() method in the file Secure.java found in hg/crawler/src/com/marketplace/io/Secure.java. 

I just modified credentials in following method on my machine.

public Secure() {
   preferences.put("username_key","<yourEmail>@gmail.com");                  
   preferences.put("password_key", "<Password>");
 }


7) It really took me a few hours to figure out how to build crawler project using 'ant' and what changes I need to do in this project before making a build.

Anyway, on my computer, path for 'crawler' folder is 'E:\tempCrawler\crawler'. Inside this folder, there is one file called 'build.xml'. Open this file and find this tag

<target name="dist" depends="compile" description="generate the distribution"> 

Under this tag, you will find one tag

<fileset dir="/Users/raunak/Development/android-marketplace-crawler/crawler/bin"/>

- Change this tag as below:

 (You will change it according to your folder path. My folder path is ''E:\tempCrawler\crawler'' and note the forward slash in the path)

<fileset dir="/tempCrawler/crawler"/>

8) Now open a command window by typing 'cmd' in the 'Start' search bar. Change your current directory to ''E:\tempCrawler\crawler'' so that command window looks like:

E:\tempCrawler\crawler>

9) Type 'ant all' without quotes on command window. It will create a few folders as given in build.xml file. Important one is "E:\Crawler\crawler\dist\lib" where ant will put 'crawler.jar' file.

If you can find this 'crawler.jar' file, that means, you have successfully built this project.

10) Now copy 'permission' file given in 'crawler' folder to 'E:\tempCrawler\crawler\dist\lib'.

Now in the 'lib' folder, you will have 'crawler.jar' and 'permission' files.

11) Open 'permission' file with your favorite text editor (I use Notepad++) and delete empty lines given at the end. Make sure there is no empty line at the end of the file.

If you don't do this, it will throw some exception and you will not be able to crawl apps from Android market.

12) Here comes the 3rd installation requirement: evil 'Ruby on Rails 3' which took my 30 hours to figure out the correct versions of Ruby and Rails for successful installation. And this is the real reason behind writing this manual.

 Download Ruby 1.9.2 from http://rubyinstaller.org/downloads/ and install it.

In my computer, it is at  "C:\Ruby192". You can install anywhere in C:\ but remember, there should not be any space in any folder name in the path.

-Type 'ruby -v' on command window and you will get output like:

ruby 1.9.2p290 (2011-07-09) [i386-mingw32]

13) Download DevKit from 'https://github.com/oneclick/rubyinstaller/downloads/'.

If you click on 'Download DevKit' , you will be able to download the setup, However, if that link is broken, download this version 'DevKit-tdm-32-4.5.2-20110712-1620-sfx.exe'.
 
During installation, it will ask where to Extract. I extracted it at 'C:\Ruby192\Devkit'. It may work in  other locations also, but I didn't check.

Post-installation instructions are given at this link. Follow it carefully.

14) After this, I am not sure what are the exact steps I followed. I did a few random installations whatever were suggested on internet, desperately wanting it to be working. However, I would try my best to mention everything required for successful installation.


I didn't have git installation on my computer, so I removed a few things in files related to git. I am not sure, if you have already installed git on your computer, it would work as it is or not.

Now go to 'server' folder at 'E:\tempCrawler\'. Open 'Gemfile.lock' file and remove first few lines so that your first two lines look like:

GEM
  remote: http://rubygems.org/

15) Open file 'Gemfile'  in server folder and find this line

gem 'vestal_versions' , :git => 'git://github.com/adamcooper/vestal_versions'

Comment the git part so that it looks like:

gem 'vestal_versions' #, :git => 'git://github.com/adamcooper/vestal_versions'


16) Add one line in 'Gemfile' file

gem "rake", "0.8.7"

17) On the command window, type 'gem install rails -v=3.0.3' as below. Note that you are in ‘server’ folder on command prompt


E:\tempCrawler\server>gem install rails -v=3.0.3

I only got 1 gem installed as I have already installed all gems, you may get more than one.

Type 'rails  --version' and it should show 'Rails 3.0.3'.


18) I forgot to mention about MySql server installation. Install mysql-5.5.28-winx64.msi .

I assume you didn't provide any password for 'root' otherwise you should note down password for root.

19) Now type 'bundle install' on the command windows. Make sure you are still in the server folder on the window.

E:\tempCrawler\server>bundle install

After successfully completing above steps, it should look like

20) Now type 'rake db:migrate' on the command window. Make sure you are still in the server folder on the window.

Since I already had created database instances given in the file 'E:\tempCrawler\server\config\database.yml' through this command, I was not able to execute this command successfully for this guide.

Note that if you provided password for 'root', you should mention password in database.xml file otherwise leave it as it.

I assume, it worked successfully on your machine otherwise, StackOverflow is best place to ask questions.

21) Now type 'http://localhost:3000/' in your browser and it should not show any error at least. Either it should show about Ruby or it should some thing like below. If you see the later one, then Congratulations.

Listing apps

Title	Creator	Category	Price

22) Now if you want to crawl apps via category, keep your server running. Open a new command window by typing 'cmd'.

Remember, once upon a time, we built 'crawler.jar' in 'E:\tempCrawler\crawler\dist\lib' folder. On command window, go to 'lib' folder and type:

java -jar crawler.jar -c

You can type Ctr+C after some time if you don't want to crawl all the apps. In the browser, hit the same URL again and you would be able to see apps information on the page.

Android-Marketplace-Crawler is best place to know what can you do with this installation.

Good Luck =).

If you like my post and want to say thanks, you can get me a coffee. =)

Downloading APK from Google Play using APK Downloader

After spending two consecutive days, I was finally able to download one APK from Google Play using APK Downloader and Google Chrome Portable V20.

Note: If you know a little bit of programming, you can use this link to download multiple APKs from Googe Play.

Here are a few steps I took to make sure I get everything working.

1)  Download Google Chrome from the following link

http://www.finaldown.com/download/software/browser-plugin/chrome/20.0.1132.47/20.0.1132.47_chrome_installer.exe

2)  Download 'APK Downloader extension' from the following link and extract it on your computer.


forum.xda-developers.com/attachment.php?attachmentid=1260350

3) Download 'Device ID' application from Google Play on your device. For reference, I am providing following link.

If you don't want to download this app, there is one trick to get device ID on your phone which is mentioned in step 8.

https://play.google.com/store/apps/details?id=com.langlearner.deviceid

4) In step 1, we downloaded chrome.exe file. Now create a shortcut for this exe file by right clicking and clicking on 'Send To' menu item. Now on shortcut icon,

right click and it will open 'Shortcut' tab by default. Against 'Target' field, your values should be like this. Don't copy paste the complete line.

<PathOnYourPC>\20.0.1132.47_chrome_installer.exe --ignore-certificate-errors --allow-running-insecure-content

5) Now open Chrome browser from this newly created shortcut. Go to 'Settings'

6) Click on 'Extensions' on the left side. Click on 'Developer mode' , then click on 'Load unpacked extension' and give path to extracted APK Downloader folder and you will see uploaded APK Downloader extension in your browser.

7) Now click on 'Options'. You might see 'Options.html' against 'background.html'. You will see following page.

 8) Now run 'Device ID' app on your device and get device ID.

However, if you don't have this application, type '*#*#8255#*#*' without quotes on your phone and against 'Device Id', you will find 'android-<device id>'. Only <device id> part is your device ID.

Enter this information on that page and you will see following page.

9) After being logged into above page, search any application on Google Play even if it's not compatible with your device. 

When you go to application page, you will see download icon against browser URL as shown below.  Click on it and application will be downloaded.

10) Rename this downloaded file from 'Download' to '<appName>.apk'

Troubleshooting

1) After making these changes, I was still not able to download APK files. It was giving me error "ERROR: - Please disable SSL error warnings" and I was not sure what I was doing wrong. Before this error and troubleshooting, I was getting error like 'Error: file can't be downloaded'.


Also, when I clicked on 'Shortcut of Chrome' we created in step 4, it was not opening chrome browser. So I did following two things before I was able to open the browser and download APK.

i) Follow the given link and make required changes to registry. I actually deleted all the keys

Make changes to Registry Keys

ii)  Check if you can delete 'Application' folder given at 'C:\Users\<UserName>\AppData\Local\Google\Chrome\'.

You might not see 'AppData' folder in <UserName> folder. Enable viewing of hidden folders. Press 'Alt' key on your keyboard, Go to Tools => Folder Options => View => Show hidden files ...

If you are not able to delete this 'Application' folder, follow the below given steps.

Restart your computer and keep pressing F8 key on your keyboard to enter into Safe Mode. Once you are logged in Safe Mode, go to above given folder again. If you are not able to find Application folder there, just create a new folder with the same name. I also deleted contents of Update folder inside the Chrome folder, just in case if it creates some problem.

7) Now just make sure your step 5 is working fine. That means, you are logged in with your username, password and device ID. Now try to download APK again.

Hope everything works fine at your computer too. =)



Due credit to following links for the help.

http://apps.evozi.com/apk-downloader/
forum.xda-developers.com/attachment.php?attachmentid=1260350
http://forum.xda-developers.com/showthread.php?t=1809458



If you like my post, and want to say thanks, you can get me a coffee. =)