Blogs

Hi, I am Malay. I am a generalist. I build whatever is required, and solve whatever problem comes my way. I reason from first principles and think broadly. Note: This document is not static. It will evolve with me. My values Following are a few principles I operate with. I try to stick to these in every aspect of my life Curiosity beats expertise I believe if you are curious you can figure out anything, and learn anything. Experts are only valuable in super specific problems and these problems are rare. I believe if I focus on asking the right questions I can find out the answers myself or I can find out the expert that I need to ask for help. Either way, being curious beats being an expert in a specific thing. ...

Why are you not productive? All you will need to do is ask around a bit and you will find that everyone works differently and has different ways of getting things done. But, If you look at how most productive people work, you will see common patterns. Some of these patterns emerge because these productive individuals are addressing, in their approach to work, a set of problems that all of us face. This is a discussion on a set of tools discovered by observing work style of such individuals, that you can implement in your daily work and get things done without exhausting yourself in the process. ...

Intro This is the 3rd part of the Low level design series. Where I will introduce you to design pattern. In this post we will talk about design patterns that I found most useful because they address some very common problems. So following are a discussion on design pattern with 6 design patterns you can start using right away. Prerequisite Design patterns are instantiations of SOLID principles in one way or another. Think of SOLIDs as specifications and design patterns are their implementation. Therefore, before reading this you should definitely read THIS article on solid principles. ...

Preface Cryptography is study of ways to convert a plain text to some unintelligible gibberish which can be converted back to the original plain text by only those who have some kind of secret knowledge. The classical analogy of cryptography goes as follows: There are three people Alice, Bob and Eve. Alice and Bob wants to exchange secret messages but Eve is always eavesdropping and intercepting their messages. To prevent Eve from reading their texts Alice and Bob decided to meet and share amongst themselves two identical keys to a padlock. Alice and Bob keep their keys a secret. Now Alice writes a message, puts it into a box, locks the box with the padlock and sends it over to Bob. Eve intercepts the box, but cannot read the message inside the box because she don’t have a key to unlock it. So she forwards that box to Bob. Bob having the key to the padlock can unlock the box and can read the message. Using this setup Alice and Bob can share secret messages without Eve reading them. ...

TL;DR 1. Use design pattern only when you are absolutely sure about the problem it will solve. 2. Goal of design is to lay the abstractions of a system to reflect the problem space and hide implementation details under appropriate abstractions. 3. SOLID are guide lines to create coherent systems. 4. SRP: Function/class/module should do one thing and do it well 5. OCP: Modules should be open for extension but closed for modification. ( A bit confusing, so read it) 6. LSP: How to not misuse inheritance: Users of base class should not notice if you pass them an object of derived class. 7. ISP: Users should only conform to the part of interface that they use. 8. DIP: This is the most important one. It tells you how to decouple a system. It says depend of abstraction rather than concretion. ...

TL;DR Mistakes to avoid Shiping the code as soon as it works Assuming requirement of an abstraction to account for probable obscure use cases. Not slowing down in development process. ( A idea on how to slow down effectitely is mentioned below) Not knowing what cleaning the code means. ( Cleaning the code is neither adding comments nor renaming variables with longer names) Few Ideas to understand to avoid the mistakes 1. Complicated vs Complex systems: Complicated puts high cognitive load, Complex is big but it makes sense. 2. Slowdown in development phase to think about maintenance to make maintenance easy 3. Design is an iterative process. You switch between development and then design with every significant change to the system. 4. How to slow down: Go fast, implement –> Slowdown, create proper abstraction to model problem space 5. How to create proper abstraction: Integrate thinking framework of SOLID principles when thinking about abstractions of a system 6. A brief introduction to SOLID principles. ...

TL;DR 1. Use ‘Elements’ tab of Chrome Dev tools to find out which element in the webpage has your data. 2. Avoid getting banned while developing by writing responses of target site in a file and using it for development. 3. Use ‘Network’ tab of Chrome Dev tools to find out structures of API calls and mimic an actual browser. 4. Parse HTML in dynamic sites only if there is no other way to do it. Try to figure out the API first. 5. Use regex to extract small pieces of information, don’t parse HTML for it. 6. Don’t overwhelm the API. 7. If you have to send a ton of requests, use proxies and shuffle user agents. (A nifty trick to use proxies mentioned on below) 8. If data is behind login pages, put wrong password once, capture the structure of the login request and use request.Session objects to login. 9. Use threadpool executor with proxies to send requests in parallel 10. Try interfacing with the mobile version of the same site if desktop version is too complicated. 11. Avoid using Selenium. Try to find out how the frontend of the APP is talking to the backend. Perform and action on the UI and capture the series of request that facilities the action using the Network tab. ...