How to create and save file(s) from a Heroku bash session

I started using Heroku for one of the projects recently. In past, I have used Openshift, Webfaction (and AWS long time ago). Each of these gave some kind of ssh access. By ssh access, I mean access to the running system.

Heroku is different.

Heroku gives you bash access. At first, it may seem similar to the ssh access, but it is not. Each time you run heroku run bash --app my_app Heroku creates a new instance of the shell. The filesystem is ephemeral i.e. if you have two such bash shells are running, and if you write something to /tmp from one instance, it won't be visible to the other bash instance.

Since there is no "instance" running (at least in a traditional sense) one can't send or receive files from the application "instance"

The only way to "send" files to application instance is to have those files committed to git and push them to Heroku. After all, that is how you update the code on Heroku.

Now, that's a problem.

Create Files

One might ask "why" would I want to send files to Heroku which are not in git

Here is an example:

I have scripts that create some artifacts by reading the CSV file as an input. I may need to run these scripts from time to time with different CSV files each time. The input files are temporary in nature. They may also contain sensitive information which I do not want to commit to git

Normally, I would ftp the files to /tmp of the machine, ssh to the instance and run the script by pointing it to /tmp/myfile.csv

As explained earlier, this does not work for Heroku

So I created a support ticket with Heroku, and here is an option they suggested.

First "login" to Heroku via heroku run bash --app my_app

$ mkdir CSV_files
$ cd CSV_files
$ cat <<'EOF' > myfile.csv
first_name, last_name
John, Doe
Jane, Doe
Mickey, Mouse
Minnie Mouse
EOF
$ cd
$ python scripts/script1.py -i CSV_files/myfile.csv

This is not the most optimal way.

I run cat myfile.csv on my local machine, and then copy/paste to the Heroku terminal session, after the cat line

Finally, I type in EOF manually, to finish the input.

This is not the most optimal solution, e.g. for long files, this could be a problem.

While there could be side effects due to how terminals handle "copy/paste", this seems to be the only option if one doesn't want the files in git

Save Files

Now that I am able to run the script on Heroku, I need to save the output from the script. Most of the times, this is not a problem. I use the "copy/paste from the terminal" But sometimes, the output is so long that it scrolls past and does not fit the terminal window's buffer.

Normally, I would redirect the output to a file (in /tmp) and FTP off the server, but again, it doesn't work with Heroku.

I came across this post on SO.

So there is this service that would accept files from anyone, anywhere, called transfer.sh. You can check the details on their site.

I was a bit hesitant letting someone-I-don't-know, host my files, but it was only the log output, that did not contain any sensitive data.

If you have any sensitive data, you can encrypt it, before uploading

You can also control the duration for which your files are accessible. The default is 14 days.

I understand that neither of these solutions is optimal, but they work in a pinch.


Flesch-Kincaid reading ease score: 93.73

Flesch-Kincaid grade level score: 2.98

Tools for removing "errors" from your writing

Initially, I was going to title this either "How to improve your writing", or "How to write better". But both those are subjective. The current title is apter [1] , I think. In technical terms, these are errors/warnings in the prose. Errors you must fix. Warnings you should fix, but can choose to ignore.

Since the "prose" isn't compiled, there is no compiler. This is similar to various interpreted (programming) languages, where there are no compilers, hence there is no easy way to catch the errors/warnings before hand. In comes the "linters" - they "can" check for potential errors before hand, and it is always a good practice to run the code via the "linter".

The prose is the same. In fact, one of the tools is aptly named exactly that proselint

So there are some of the tools I have come across recently.

Grammarly

As the name suggests, it is mainly used for checking the Grammar. Useful for someone like myself for whom English is not the first language.

All of my writing is in emacs. So Grammarly does not directly integrate with my workflow. There is a native app, where I can copy paste this text and make corrections, and import the fixes "back" to emacs :) Bit tedious if you ask me.

write-good-mode

I've already written about it earlier. It is a minor mode in emacs. It identifies weasel words (words that do not add value or change the meaning), passive voice, duplicate words.

proselint

This is a python program. you install it using pip install proselint

Apparently, there is a way to integrate this with emacs as well (Need to research)

Here is an output from proselint when run on one of my previous post [2]

$ proselint ../posts/gitlab-project-not-found.rst
../posts/gitlab-project-not-found.rst:8:55: cliches.write_good 'Needless to say' is a cliché.
../posts/gitlab-project-not-found.rst:33:36: typography.symbols.ellipsis '...' is an approximation, use the ellipsis symbol '…'.
../posts/gitlab-project-not-found.rst:55:29: hyperbolic.misc '!!!' is hyperbolic.
../posts/gitlab-project-not-found.rst:55:30: leonard.exclamation.multiple Stop yelling. Keep your exclamation points under control.

When I ran it against some other posts, there was no output, indicating that those posts were "fine" :)


Footnotes:

[1] I had chosen to write "more apt". Grammarly suggested "apter". I had never heard of this word. I had to check to see whether this was a false report. It wasn't.
[2] I've since fixed 3 out of these 4 errors. The error about ellipsis is verbatim output from ssh-add --help

GitLab: The project you were looking for could not be found

Every now and then I used to keep getting this error. (The project existed.) This error showed up once in a while. My go to solution was to reboot the machine (cause it used to work) But for once, that too did not work, and it led me to finding the real cause /solution. I'm glad I did.

I had noticed that I get this error when SourceTree asks for passphrase for some other publickey. But it wasn't always the case.

As normal troubleshooting tips, I tried GIT_SSH_COMMAND="ssh -v" which showed that the correct key was in use. This did not surprise me because by ~/.ssh/config was correct (Otherwise I would ALWAYS have problem accessing gitlab, which was not the case)

Whenever I rebooted and used any git command afresh, I used to see message "Identity Saved" (after I entered the correct pass phrase). I had a hunch that somehow my problem was somehow related to this. I still don't know what is the problem, but I came across the solution.

Turns out way to fix this is remove the identity using ssh-add command.

$ ssh-add --help
ssh-add: illegal option -- -
usage: ssh-add [options] [file ...]
Options:
  -l          List fingerprints of all identities.
  -E hash     Specify hash algorithm used for fingerprints.
  -L          List public key parameters of all identities.
  -k          Load only keys and not certificates.
  -c          Require confirmation to sign using identities
  -t life     Set lifetime (in seconds) when adding identities.
  -d          Delete identity.
  -D          Delete all identities.
  -x          Lock agent.
  -X          Unlock agent.
  -s pkcs11   Add keys from PKCS#11 provider.
  -e pkcs11   Remove keys provided by PKCS#11 provider.
  -A          Add all identities stored in your keychain.
  -K          Store passphrases in your keychain.
              With -d, remove passphrases from your keychain.

I still need to investigate how to remove a specific identity (I didn't spend much time. I just have two entries saved. I removed them both. Not a bug deal)

Once I removed the saved identities, git command asked for the pass phrase, and gitlab was accessible again.


Flesch-Kincaid grade level score: 2.40

Flesch-Kincaid reading ease score: 91.67

Mindfulness Course : Week 1

I started an online course called "Mindfullness for Wellbeing and Peak Performance" at Monash University on FutureLearn. (Name of the course is a mouthful, hence I decided to shorten the name as referenced in this post) These are my notes from Week 1


Mindfulness means being engaged and present in the moment. We are usually mindful during the activities that we enjoy. When we are not happy is when our mind wanders off (to worry, thinking about the past or the future) and so that is when we need to be mindful.

Notice that mind has wandered off, and bring it back to the present moment.

When we are not mindful, we are in default mode, where the mind wanders, mostly thinking about past or future, causing stress (and not being productive)

Attention Deficit Trait : When we multi-task, we are working on one thing and thinking on five other things. Or getting distracted by the phone call, or knock on the door. It activates amygdala so much that we're getting to a sort of a chronic fight or flight response

Multitasking is an illusion

Attentional Blink : quick pause when we switch the context. During this short 0.2=0.5 seconds, we are not paying attention to anything at all.

To practice mindfulness, we need to manage the environment as well (Turn off all the alerts on the computer and/or phone)

Stress performance curve : Zero stress may mean we are apathetic, little bit of stress is good, motivates us (Top of the bell curve) and more stress, then reduces the performance.

Mindfulness helps reduce the stress, but that doesn't mean it impacts performance (based on the stress performance curve) rather when we are really focused, we are "in the zone" (or flow state) and have highest performance.

allostatic load : when we’re activating the stress response when we don’t need it, it produces a wear and tear on our system that’s called allostatic load. This causes immune problem Cardiovascular problems etc. mindfulness is that it helps to switch off the inappropriate activation of that stress response and takes off a lot of that allostatic load.

stress actually accelerates the ageing on the level of the DNA and mindfulness has even been found to actually start to improve genetic repair and slow down that ageing process.

Mindfulness Journal : Write about mindful experiences and unmindful ones too. The aim of this is learning, not criticising ourselves for not getting it right. Mindfulness is about being curious and accepting, rather than judgemental, and your journal should reflect this approach.

Ulysses Pact

While listening to the Changelog Episode 221, where they interview Dr. Cory Doctorow. In the Interview, Doctorow mentioned Ulysses Pact.

In the Greek mythology, it is said that whenever ship passes near the Siren's island, the men would hear Siren's Song. upon which they would lose their rational thought and jump the ship to swim towards the Siren's island. (Sirens would kill the men)

Ulysses wanted to hear the Siren's Song, but he also knew the risks. So he ordered his men to tie him to the mast of the ship so that he could not jump the ship (when he will hear the Siren's song and go insane) He also ordered his men to put wax in their ears so that none of them could hear the song, and all of them would remain "sane" (and continue on the course they had planned earlier)

This story drives the point that one can "protect oneself" from future risks by planning for them now. To expand further on this, this "advanced" decision is taken when the person is of "strong" mind and recognizes that in future may become weak, and act in a way that may be harmful.

e.g. When on a diet, instead of trying to "fight" against eating an "Oreo" cookie (in a weak moment) It is better to "Not to buy the Oreo at all" and/or "Throw away the ones you have" (when you are strong)

More recent example is that starting with iOS 8, even apple can not unlock the user's iphone data (Only the user can) This means even if a government body forces them to get the user's data, they (Apple) can't.


References :

Prevent *scratch* buffer from getting killed

I started using persistent *scratch* buffer few months ago. You can read about it here

It worked well for some time, and then at least couple of times, I noticed that when I restart emacs (does not happen very often) my *scratch* buffer is empty.

So I started doing M-x persistent-scratch-save every time I was done writing to *scratch*, even though persistent-scratch-autosave-mode was ON

Just recently I realized that I had "lost" my *scratch* buffer while still within my emacs session (It did not show up in the buffer list) So then I realized why I might be losing the contents. It is my guess that when exiting persistent-scratch will save the contents to disk before exiting, but there was no *scratch* buffer so it creates empty file.

I was under the impression that all the buffers starting with * were special buffers and thus can't be killed - guess I was wrong. I came across the solution on Stack overflow

I'm listing the relevant code here for your reference :

;; bury *scratch* buffer instead of kill it
;; http://stackoverflow.com/a/358740/154947
(defadvice kill-buffer (around kill-buffer-around-advice activate)
  (let ((buffer-to-kill (ad-get-arg 0)))
    (if (equal buffer-to-kill (get-buffer-create "*scratch*"))
        (bury-buffer)
      ad-do-it)))

QOD : Deliberate Practice

I am a regular "listener" (?) of Freakonomics and "James Altucher Show" podcasts among others. So when Steve Dubner and James Altucher started a podcast together there was no reason to not subscribe. Needless to say, their new show "QOD : Question of the Day" is both entertaining and full of information.

This post is related to their episode no. 118 about deliberate practice. You can listen to the episode here.

Better still, why not subscribe to the podcast ?

Couple of interesting stories from this podcast ..

London Taxi Drivers Vs London Bus Drivers

Both taxi drivers and bus drivers, drive vehicles (for other people) full time, so in a sense their job is alike. It was expected that after 30 years of career in their respective jobs, the brain development for both would be similar. Turns out, brain of the taxi driver evolved "better" than that of the bus driver. Dubner goes on to explain why.

Taxi driver has to learn new route for each new passenger they carry (at least first few years anyway) Bus drivers on the other hand have "fixed" route, even if they have rotation every few weeks. So taxi driver has to constantly practice, while bus driver doesn't. Hence the difference between the growth of their brain.

General Practitioner

Dubner comments that after years of "General Practice", brain of a doctor remains at same level as the time they graduate.

When they are students, they are learning a LOT. Both via books and experience. But once they start the practice, it is not up to them to "choose" type patients they treat.

Doctor "dignoses" the patient, prescribes the medicine, and that is the end of it. Patient may not return for various reasons. May be the medicine "worked", may be they went to another doctor, may be patient died !! (Yes, they say that on the podcast) May be the patient would have been cured on their own.

There is no way to tell (unless patient returns for the follow up)

James says that the feedback is not enough (in the above case)

Mentor

They talk about how "quick enough" feedback is important. Sometime (like in formal education) feedback is easy in the form of test scores. But in some cases it is difficult. e.g. "writing", or other creative art. You may not even get any feedback EVER. or not enough, or not on time. Also the feedback might be subjective and may not be directly useful. "I liked (or did not like) your painting" - How do you use such feedback ?

Mentor can be helpful in such cases.

Aside

Not covered in the podcast, but the subject of "Deliberate practice" also came up in the coursera course "Learning how to learn". They say that attempting varied and/or (perceived) difficult problems can "solidify" your understanding. You not only understand "how", but also "when" to use the particular technique.

Habit Loop

I'm reading a book The Power of Habit by Charles Duhigg

So far I am fascinated by the information. Writing style is quite engaging.

In first few chapters he talks about "Habit loop"

Cue -> Routine -> Reward

.... and suddenly I started seeing "habits" everywhere

Two habits I want to mention here are of my children


A younger one, just about an year old, throws tantrums when we are changing his nappy or putting on a diaper. But I started playing peek-a-boo with him when he is lying on his back, and he closes his eyes and plays peek-a-boo since he enjoys that, he lays "still", till I put on a fresh diaper

Cue : Lay on the back watching father

Routine : Lay still/put hands on eyes and play peek-a-boo

Reward : Fun !!


His older brother is about 3 yrs old. He doesn't want to leave TV and go to sleep. Recently I showed him "shadow play" where all his toy animals appear before "torch", then their shadows do something "fun", and jump on the bed and sleep. He really enjoys this.

.... and at the end, after all the animals have "gone to sleep", he is the last one to look at his own shadow, wave, and jump on the bed to sleep.

He likes this so much, that I just have to "remind" him of the "shadow play" and he will be OK me switching off the TV

Cue: Reminder about "upcoming" shadow play

Routine: brush his teeth, a mini-bath, putting on night dress

Reward : Shadow play


Now, I need to look at my own life to see the habit-loop

Rubber Duck Problem Solving

Background

In case you are not familiar with Rubber Duck approach to solving a problem, it goes something like this :

Sometimes, if one spends enough time analyzing the problem, one can find the solution on their own. But most of the times, we don't. We get stuck and "ask" someone. Off course, that "someone" is not as familiar with the problem as you are. So in order to help you, they ask you questions (so they can understand the problem first, before they can help you) But it is often seen that when answering "them", you come up with the solution your self.

So where does the "Rubber Duck" fit in ?

The point is, "they" didn't solve the problem, "you" did. By explaining the details. So in this case "they" might as well have been a "Rubber Duck". You can explain the problem to a rubber duck, and will achieve the same effect. See this Wikipedia Entry for more details.

I have noticed several times, that right after I post a query on a mailing list, I will get "unstuck" myself (and I've posted answer to my own query - so that someone else might be helped in future)

So here is a recent example of how I came upon an answer by talking to a (metaphorical) rubber duck.

Problem

I was getting the following error on heroku

app[web.1]:   File "/app/.heroku/python/lib/python3.5/site-packages/flask_mail.py", line 566, in init_app
app[web.1]:     state = self.init_mail(app.config, app.debug, app.testing)
app[web.1]:   File "/app/.heroku/python/lib/python3.5/site-packages/flask_mail.py", line 552, in init_mail
app[web.1]:     int(config.get('MAIL_DEBUG', debug)),
app[web.1]: ValueError: invalid literal for int() with base 10: 'True'

Based on the code it refers to, the error seems correct in isolation. But flask-mail documentation says (and the previous line in the stack trace also agrees) that value of app.debug is used for MAIL_DEBUG

I had set the DEBUG as follows :

class HerokuConfig(Config):
    import os
    DEBUG = os.environ.get('DEBUG_MY_APP', False)

I've set it to True from heroku console

$ heroku config:get DEBUG_MY_APP -s
DEBUG_MY_APP=True

But it looks like this somehow gets converted to string 'True', resulting into an error

$ python
Python 3.4.3 (default, Apr 27 2015, 19:08:17)
Type "help", "copyright", "credits" or "license" for more information.
>>> int(True)
1
>>> int('True')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: 'True'
>>>

So how do I prevent the DEBUG from turning into a string ? I think there is something obvious that I am missing.

Solution

the root cause was that os.environ.get() always returns a string. So the correct way to set DEBUG is as follows :

class HerokuConfig(Config):
    import os
    DEBUG = os.environ.get('DEBUG_MY_APP', 'False') == 'True'

Flesch Kincaid grade level score : 2.50

Flesch Kincaid reading ease score : 92.08

Unlocking the secrets of meditation

This is a summary of the following youtube video

by Dandapani : Unlocking the secrets of Meditation


Understanding Meditation

  • Meditation is not a sweet sauce you add to your life to make it taste good.
    • Having a carrot in the morning, and the hamburgers, and beers through out the day is not a "Healthy" diet. Similarly, meditation in the morning is not enough (?)
  • First fix your lifestyle that supports meditation
  • What is meditation :
    • Prolonged state of concentration
    • Meditation is working with the energy inside of you
  • We can't clear your mind, but we could focus the mind.
  • Any meditation practice should have a process and a goal
  • Meditate on same thing for (at least 5 days a week) before you switch, else you'll never make progress.
  • To meditate, you need to be able to concentrate.
  • How to concentrate :
    • Do one thing for a prolonged time
    • We can not multi-task
  • Meditation takes courage (to face "ugly" self)
    • When we start meditating, we face our "ugly" subconscious
    • we never spent time "cleaning" up the subconscious
  • Takes self-acceptance
    • Accept that who you are today took years, so it will take some years to "un-create"
  • Just like diet, meditation will take long time before we see results.

Process

  • Start with "1 minute a day" (Cause it is sustainable)
  • Create a daily ritual
  • First thing in the morning
  • Don't interact with anyone (may be shower, coffee/water is fine)
  • Meeting with your energy
    • Meetings need time and location
  • Sit with spine straight (preferably without back support)
  • Left hand below, right hand on top, thumbs touching each other.
  • If/when you are not concentrating, the tips of thumbs are no longer touching and spine is no longer straight.
Google Analytics Alternative