[mlpack] Ready to use Models in mlpack (GSOC '21)

Mon Mar 29 10:54:17 EDT 2021

Hey Marcus,

So it would ideally be the same because a modular structure will allow us
to build resnet with different layers and I have gone through the source
code that PyTorch has they only create it in modules because it allows
creating different Resnets without every time writing the same
block(multiple layers), So I think as to narrowing down my proposal I would
leave segmentation, for now, I can totally do it even after GSoC and would
propose a modular resnet and if time persists I would also like to add
mobilenetv3 in a modular way because if I can start that work in the GSoC
period I will finish it even after GSoC ends.

Feeling really stocked to start working on this and as I finish iterating
over my proposal, I will submit an initial draft through GSoC so you can
take a look and reach back for feedback and help around if I forgot to
mention something because I won't like proposing a well-documented project
is no joke and the applications get big and it's hard for you to even read
them.

Best,
Aakash.

On Mon, Mar 29, 2021 at 6:58 PM Marcus Edel <marcus.edel at fu-berlin.de>
wrote:

> Agreed, ideally, the models should be customizable, but I don't think we
> have to
> provide an implementation that has the same functionalities as PyTorch,
> it's good
> to think about possible extensions, though. Also, I agree that we should
> implement
> the parts as independent blocks in the models repository. Depending on
> what you
> like to propose, it would be helpful to reduce the project's scope, like
> if you are
> planning to implement resnet with the same functionalities as PyTorch, I
> think that
> alone is enough work to fill the summer.
>
> On 28. Mar 2021, at 12:19, Aakash kaushik <kaushikaakash7539 at gmail.com>
> wrote:
>
> Hey Marcus,
>
> Thank you for reaching back, So when implementing resnet we would need to
> implement it in such a way that it supports all the versions that Pytorch
> has to offer and on top of that all the extras that are needed to use them
> as backbones amd the same goes with mobilenet we cannot kist implement a
> single flavour of them we would need to implement these models do as they
> are customisable in themselves just like Pytorch. What do you think about
> that and what do you propose i go ahead with?
>
> Best,
> Aakash
>
> On Sun, 28 Mar, 2021, 9:45 pm Marcus Edel, <marcus.edel at fu-berlin.de>
> wrote:
>
>> Hello Aakash,
>>
>> I really like the thoughts you put into the project, as you already
>> pointed
>> out for deeplab v3 we would need resnet 50, resnet 101, and MobileNet v3,
>> but looks like MobileNet is missing from the second mail timeline.
>>
>> Looking at the timeline, I'm wondering if it makes more sense to implement
>> MobileNetV3 and restnet 50 as independent modules first and maybe do
>> the rest if there is time left. That said, I guess implementing MobileNet
>> with
>> some strong test could already fill the summer.
>>
>> Let me know what you think.
>>
>> Thanks,
>> Marcus
>>
>> On 28. Mar 2021, at 11:48, Aakash kaushik <kaushikaakash7539 at gmail.com>
>> wrote:
>>
>> Slight changes to the previous email I would like to mention in my
>> proposal that the aim will be to try to extract the weights from PyTorch
>> and make the models pretrained and not just predefined using the converter
>> but I am not 100% sure if it would work so will that be something
>> acceptable?
>>
>> On Sun, Mar 28, 2021 at 8:44 PM Aakash kaushik <
>> kaushikaakash7539 at gmail.com> wrote:
>>
>>> Adding to the previous email it should be easily possible to add both
>>> the version of deeplab_v3 which are with the resnet 50 and resnet 101
>>> backbone because they have the same blocks but just different params with
>>> the different number of layers so I want to discuss the points mentioned in
>>> the previous mail but also to add to that I want to discuss the scope of
>>> the project should I keep it concretely to a data loader and to predefined
>>> but not pre-trained models which are deeplab_v3 with resnet 50 and resnet
>>> 101 backbone.
>>> For an iteration, I would like to propose adding the following to just
>>> make it clear.
>>> 1. segmentation data loader
>>> 2. deeplabv3 (resnet50 and resnet101) predefined and not pre-trained
>>>
>>> I know we previously talked about adding another model if time permits,
>>> but I am not really sure about that, so I wanted to discuss these and see
>>> if I should add the third model to my proposal and mention the time
>>> constraint and just let it be a potential model or if I should completely
>>> remove the idea of the third model.
>>>
>>> Best,
>>> Aakash
>>>
>>> On Sun, Mar 28, 2021 at 2:33 PM Aakash kaushik <
>>> kaushikaakash7539 at gmail.com> wrote:
>>>
>>>> Hey Everyone,
>>>>
>>>> This is a continued mail regarding my proposal and I have been taking a
>>>> deeper look and found out that PyTorch implements these segmentations
>>>> models above their pre-existing pipelines of backbones and I proposed to
>>>> implement 3 things 1 data loader for segmentation task and 2 models from
>>>> which one would be a potential model only if time permits but the way I see
>>>> for the present time they can be implemented in mlpack is by creating block
>>>> and creating completed models because no such customizable backbones exist
>>>> as of now that can be called. and so this opens another question as to such
>>>> deeplab_v3 consists of three backbones in PyTorch which are resnet 50,
>>>> resnet 101, and mobilenet v3 large and as coco is such a huge dataset and
>>>> the tool for converting weights from torch to mlpack is a bit flack should
>>>> I go with a proposal to include predefined models which can be trained
>>>> rather than pre-trained models for now and if it permits we can add the
>>>> weights for coco in the future. I think these were some of the doubts I had
>>>> while I was writing the exacts of my proposal and I will be glad to have a
>>>> discussion on it in terms of how concrete it is and if you guys see a
>>>> problem on how they can be implemented or any other doubts or questions.
>>>>
>>>> Best,
>>>> Aakash
>>>>
>>>> On Thu, Mar 18, 2021 at 5:35 AM Aakash kaushik <
>>>> kaushikaakash7539 at gmail.com> wrote:
>>>>
>>>>> Hey Marcus,
>>>>>
>>>>> I totally got it and i think 1 data loader and 2 models from which 1
>>>>> will be a potential model if only time permits.
>>>>>
>>>>> Thank you for the feedback and help. :D
>>>>>
>>>>> Best,
>>>>> Aakash
>>>>>
>>>>>
>>>>> On Wed, 17 Mar, 2021, 9:33 pm Marcus Edel, <marcus.edel at fu-berlin.de>
>>>>> wrote:
>>>>>
>>>>>> Yes, that’s what I had in mine, but at the end it’s your decision.
>>>>>> About the
>>>>>> model either is fine, you can select whatever you find more
>>>>>> interesting.
>>>>>>
>>>>>> On 17. Mar 2021, at 10:45, Aakash kaushik <
>>>>>> kaushikaakash7539 at gmail.com> wrote:
>>>>>>
>>>>>> HI Marcus,
>>>>>>
>>>>>> Thank you so much for reaching back, So just to clarify i would keep
>>>>>> the deliverables to just two which will be:
>>>>>>
>>>>>> 1. Semantic segmentation dataloader in the format of COCO dataset .
>>>>>> 2. One semantic segmentation model
>>>>>>
>>>>>> If I understood you correctly, will you be able to help me decide
>>>>>> which kind of model I should add, should i go for a model that is more
>>>>>> generally used such as U-Net or one from the above list that PyTorch has ?
>>>>>>
>>>>>> Best,
>>>>>> Aakash
>>>>>>
>>>>>> On Wed, Mar 17, 2021 at 7:55 PM Marcus Edel <marcus.edel at fu-berlin.de>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello Aakash,
>>>>>>>
>>>>>>> thanks for the interest in the project and all the contributions;
>>>>>>> what you proposed
>>>>>>> looks quite useful to me and as you already pointed out would
>>>>>>> integrate really well
>>>>>>> with some of the existing functionalities.
>>>>>>>
>>>>>>> I guess for loading segmentation datasets we will stick with a
>>>>>>> common format e.g.
>>>>>>> COCO, and add support for the data loader and potentially add
>>>>>>> support for other
>>>>>>> formats later?
>>>>>>>
>>>>>>> One remark about the scope, you might want to remove one model from
>>>>>>> the list, and
>>>>>>> add a note to the proposal something along the lines of, if there is
>>>>>>> time left at the end
>>>>>>> of the summer, I propose to work on z, but the focus is on x and y.
>>>>>>>
>>>>>>> I hope what I said was useful; please don't hesitate to ask if
>>>>>>> anything needs clarification.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Marcus
>>>>>>>
>>>>>>> On 16. Mar 2021, at 00:16, Aakash kaushik <
>>>>>>> kaushikaakash7539 at gmail.com> wrote:
>>>>>>>
>>>>>>> Hey everyone,
>>>>>>>
>>>>>>> My name is Aakash Kaushik <https://github.com/Aakash-kaushik> and I
>>>>>>> have been contributing for some time specifically on the ANN codebase in
>>>>>>> mlpack.
>>>>>>>
>>>>>>> And the project idea that is ready to use Models in mlpack peaks my
>>>>>>> interest. So initially i would like to propose a data loader and 2 models
>>>>>>> for semantic segmentation because i see that the data loaders for image
>>>>>>> classification and object detection are already there and including a
>>>>>>> couple of models and a data loader in GSOC for semantic segmentation will
>>>>>>> open the gates for further contribution of models in all three fields as
>>>>>>> they would only need to worry about the model and not loading the data and
>>>>>>> also will have some reference models in that field
>>>>>>>
>>>>>>> So the data loader would be capable of taking up image segmentation
>>>>>>> data that is the real image, segmentation map, segmentation map to class
>>>>>>> mapping, and for the models i am a bit confused as if we want some basic
>>>>>>> nets such as U-nets or a combination of both a basic net and state of the
>>>>>>> art model, or two state of the art model. Pytorch supports couple of models
>>>>>>> in the semantic segmentation fields which are:
>>>>>>>
>>>>>>> 1. FCN ResNet50, ResNet101
>>>>>>> 2. DeepLabV3 ResNet50, ResNet101, MobileNetV3-Large
>>>>>>> 3. LR-ASPP MobileNetV3-Large
>>>>>>>
>>>>>>> And so i should be able to convert their weights from pytorch to
>>>>>>> mlpack by modifying the utility created by kartik dutt which is
>>>>>>> mlpack-PyTorch-Weight-Translator
>>>>>>> <https://github.com/kartikdutt18/mlpack-PyTorch-Weight-Translator>
>>>>>>>
>>>>>>> I am trying to keep the deliverables to just three which is a data
>>>>>>> loader and 2 models as the GSOC period is reduced to just 1.5 months and
>>>>>>> for these three things i would have to write tests, documentation and
>>>>>>> example usage in the example repository.
>>>>>>>
>>>>>>> And before this work as we are in the process of removing boost
>>>>>>> visitors from the ANN codebase and had couple of major changes to the
>>>>>>> mlpack codebase the models repo wasn't able to keep up with it so my main
>>>>>>> goal before GSOC starts would be to work on the PR that is to  Swap
>>>>>>> boost::variant with vtable
>>>>>>> <https://github.com/mlpack/mlpack/pull/2777> and then make changes
>>>>>>> to the code in models repo to adjust the change in boost
>>>>>>> visitors, serialization and porting tests to catch2.
>>>>>>>
>>>>>>> I wanted to hear from you if this is the right path and if the
>>>>>>> number of deliverables are right for this and help in choosing the exact
>>>>>>> models that i should pick that would be the most helpful or beneficial to
>>>>>>> the library.
>>>>>>>
>>>>>>> Best,
>>>>>>> Aakash
>>>>>>> _______________________________________________
>>>>>>> mlpack mailing list
>>>>>>> mlpack at lists.mlpack.org
>>>>>>> http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://knife.lugatgt.org/pipermail/mlpack/attachments/20210329/76e8c0fe/attachment-0001.htm>