dallemini module v1 by hexunlin · Pull Request #416 · open-mmlab/mmgeneration

hexunlin · 2022-08-30T20:34:11Z

dallemini module v1 + vqgan module v1

dallemini modules v1

vqgan modules v1

update format2

update_format3

update format4

update format

update format6

plyfager · 2022-09-07T05:41:11Z

mmgen/models/architectures/dalle_mini/modules.py

+
+
+@MODULES.register_module()
+class GLU(nn.Module):


May nn.GLU meets our needs?

plyfager · 2022-09-07T05:42:06Z

mmgen/models/architectures/dalle_mini/modules.py

+
+
+@MODULES.register_module()
+class EncoderLayer(nn.Module):


Since this encoder layer is used for Bart, BartEncoderLayer may be a better name.

plyfager · 2022-09-07T05:42:25Z

mmgen/models/architectures/dalle_mini/modules.py

+
+
+@MODULES.register_module()
+class DecoderLayer(nn.Module):


Since this decoder layer is used for Bart, BartDecoderLayer may be a better name.

plyfager · 2022-09-07T05:44:39Z

mmgen/models/architectures/dalle_mini/modules.py

+
+
+@MODULES.register_module()
+class AttentionBase(nn.Module):


May this module meets our needs?

plyfager · 2022-09-07T05:45:10Z

mmgen/models/architectures/dalle_mini/modules.py

+
+    def __init__(self, in_out_channels, mid_channels):
+        super().__init__()
+        self.norm1 = build_norm_layer(dict(type='LN'), in_out_channels)[1]


_, self.norm1 = build_norm_layer() seems better

plyfager · 2022-09-07T05:46:40Z

mmgen/models/architectures/dalle_mini/modules.py

+
+    def __init__(self, in_channels, head_num, out_channels):
+        super().__init__()
+        self.selfAttention = AttentionBase(in_channels, head_num)


maybe we can just name it self.attn.

plyfager · 2022-09-07T05:51:14Z

mmgen/models/architectures/dalle_mini/modules.py

+        x = self.selfAttention(q, k, v, attention_mask)
+        x = self.norm(x)
+        x = residual + x
+        residual = x.clone()


In fact you can just write code in this way

h = self.glu(x) x = h + x

instead of

residual = x.clone() x = self.glu(x) x = residual + x

plyfager · 2022-09-07T05:52:51Z

mmgen/models/architectures/dalle_mini/modules.py

+            x (torch.FloatTensor): Output feature map.
+        """
+        residual = x.clone()
+        x = self.norm(x)


write forward in this way

h = self.norm(x) h = xxx(h) x = x + h

without using clone.

plyfager · 2022-09-07T05:53:35Z

mmgen/models/architectures/dalle_mini/modules.py

+        self.crossAttention = AttentionBase(in_channels, head_num)
+        self.norm = build_norm_layer(dict(type='LN'), in_channels)[1]
+        self.glu = GLU(in_channels, out_channels)
+        self.token_indices = torch.arange(256, device=device)


You may set 256 as an argument in init function.

plyfager · 2022-09-07T05:58:34Z

mmgen/models/architectures/dalle_mini/modules.py

+        in_channels (int): The channel number of the input feature map.
+        head_num (int): Number of heads in the attention.
+        out_channels (int): The channel number of the output feature map.
+        device (str): The type of device (cpu or cuda).


in fact, device is not supposed to be set in init function. MMCV or MMEngine will put model in correct device. Or you can just use model.to(device) outside.
If you really need to get the device of a model. Call get_module_device. Or you may use type_as for tensors.

plyfager · 2022-09-07T06:08:04Z

mmgen/models/architectures/vqgan/modules.py

+from mmgen.registry import MODULES
+
+
+def nonlinearity(x):


may we use nn.silu?

plyfager · 2022-09-07T06:09:12Z

mmgen/models/architectures/vqgan/modules.py

+    return x * activate(x)
+
+
+def Normalize(in_channels):


We do not need to add an extra function here. Just use build_norm_layer in your code.

norm_cfg can be set as an argument.

plyfager · 2022-09-07T06:11:04Z

mmgen/models/architectures/vqgan/modules.py

+
+
+@MODULES.register_module()
+class DiffusionDownsample(nn.Module):


I'm wondering whether we call this module DiffusionDownsample.😂

This module is a single stride2 conv or avg_pool. We may not add an extra class here.

plyfager · 2022-09-07T06:16:06Z

mmgen/models/architectures/vqgan/modules.py

+
+
+@MODULES.register_module()
+class DiffusionResnetBlock(nn.Module):


If this resblock is the same as diffusion unet, you may find it in diffusion architecture.

plyfager

MMGen have supported DDPM, you may see whether you can reuse its modules.(unet, downsample, upsample).

plyfager · 2022-09-07T06:20:01Z

For above comments, you may check whether other lines also have same problem and fix them.

Fixed format in dalle_mini and vqgan modules; Extended Downsample in ddpm; GLU can't be replaced by nn.glu(); Temporary keep AttentionBase and DiffusionResblock (needs further testing).

quantizer module for vqgan

OpenMMLab-Assistant005 · 2023-04-11T14:37:49Z

Hi @hexunlin ！We are grateful for your efforts in helping improve this open-source project during your personal time.
Welcome to join OpenMMLab Special Interest Group (SIG) private channel on Discord, where you can share your experiences, ideas, and build connections with like-minded peers. To join the SIG channel, simply message moderator— OpenMMLab on Discord or briefly share your open-source contributions in the #introductions channel and we will assist you. Look forward to seeing you there! Join us ：https://discord.gg/UjgXkPWNqA
If you have a WeChat account，welcome to join our community on WeChat. You can add our assistant ：openmmlabwx. Please add "mmsig + Github ID" as a remark when adding friends：）
Thank you again for your contribution❤

hexunlin added 2 commits August 31, 2022 04:26

dallemini_modules_v1

7396ca3

dallemini modules v1

Update __init__.py

089e022

hexunlin requested a review from plyfager August 30, 2022 20:34

hexunlin added 9 commits August 31, 2022 07:18

vqgan_modules_v1

a16319f

vqgan modules v1

Update modules.py

9066ac5

Update __init__.py

022d187

update format

cd403c4

update_format2

af256a3

update format2

update_format3

619879d

update_format3

update_format4

c689d7b

update format4

update_format5

c9b02bb

update format

update_format6

d35b897

update format6

plyfager reviewed Sep 7, 2022

View reviewed changes

hexunlin added 2 commits September 14, 2022 17:35

fixed_format_bugs

22ecc78

Fixed format in dalle_mini and vqgan modules; Extended Downsample in ddpm; GLU can't be replaced by nn.glu(); Temporary keep AttentionBase and DiffusionResblock (needs further testing).

vqvae_quantizer

7a99c9b

quantizer module for vqgan

zengyh1900 assigned plyfager Oct 12, 2022

zengyh1900 added awaiting response kind/feature request new feature/model/datasets/config etc. priority/P0 highest priority labels Oct 12, 2022

zengyh1900 added this to the Backlog milestone Oct 12, 2022

plyfager added status/WIP work in progress normally and removed awaiting response labels Oct 13, 2022



		@MODULES.register_module()
		class DiffusionDownsample(nn.Module):



		@MODULES.register_module()
		class DiffusionResnetBlock(nn.Module):

Conversation

hexunlin commented Aug 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

plyfager left a comment

Choose a reason for hiding this comment

Uh oh!

plyfager commented Sep 7, 2022

Uh oh!

OpenMMLab-Assistant005 commented Apr 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hexunlin commented Aug 30, 2022 •

edited

Loading